Is parson brown referring to the clergy or the color brown?

Parson Brown traditionally refers to the clergy, specifically a personal name often used in songs. The color brown is not typically associated with the term in this context.