What color scarf match brown shirt?

Theoretically, to actually "match," the shirt would need to be scarlet to match the scarlet shorts.

There are colors that would look good with, or "complement" scarlet, like white, black, gray, and some shades of blue, green, or pink.