How old were Jesus' brothers?

Mark 6:3 identifies Jesus' brothers as James, Joses, Juda and Simon. The nativity accounts in both Matthew and Lukeseem quite clear that when Jesus was born there was no older sibling. Jesus would have to have been the oldest.