What is a picture within a picture called?

Its called a picture within a picture

The term is inset.

A picture that has a smaller copy of itself within the picture, with the copy having an even smaller copy, that continues to (theoretically) infinity, is called a Droste Effect or recursive picture.