What is the study of x rays and radioactive substances?

The study of high-energy, electromagnetic radiation, which includes x-rays, is called atomic spectroscopy.

The study of nuclear radioactivity and decay is called nuclear physics.

For the study of electromagnetic radiation of energies below x-rays you have:

UV - UV spectroscopy

Visible Light - gaffer

Infra-red - infrared spectroscopy

Microwave - microwave spectroscopy

Radio - amateur broadcaster