That can be summed up to bad English translation. If you took the Greek writing and compared the words used for brothers of Christ it the same word we see used for cousins in the Greek Septuagint. So we can see it is a imprecise word to be translating as brothers. Now did he have cousins sure, and if the tradition is correct Joseph had children from a first wife who passed away so he would have brothers in that since. A lot of people look at there English bible and act like it feel out of the sky from God's hands. It is written in a exact moment in time, in a language very different than English from a oral tradition that existed before it. So it has to be read in light of Reason and Tradition.