Can the Children Aid Society refuse to give you your file?

Not really enough specific information is given with which to answer.

However, answering in a general manner, if you did not hire them or employ them in some manner, and they are not government agency, then you really have no right to have access to their records, or copies of their files.

On the other hand, IF you DID employ them or they owe you some service for which you paid them, - OR - they are operating as an "agent" of government you POSSIBLY could be entitled to access whatever files on you they may have.

You would have to inquire further with the authorities in your particular state.