Would you capitalize paragraph if it is referring to a specific paragraph number in a paragraph?

No, in this case "paragraph" would not be capitalized because it is not part of the proper name or title of that paragraph.