How do you hookup security camera to older TV with coaxial input only?

You need a 'modulator'. This device will accept audio and video via RCA plugs or SCART and outputs on a UHF TV channel. Some, you adjust the frequency with a screw driver, but more modern ones are digital and can select from CH 21 to 69 via an led display and switches.

A cheaper and simpler method would be to use an old VCR (maybe one that no longer plays tapes). Use the output of this to supply the TV (usually on CH35) and the AUX in to provide the inputs via the back connections.