Why can't rosaries be worn around the neck?

You can't because it is a sign of disrespect. Also it show that you don't care about the real meaning of it. You should hold it in your hands. You can have a rosary bracelet or key chain but it was made specifically for that purpose.