Is it racist to call someone a monkey?

Depends on the context.

In certain countries certain Words or phrases have a "local significance" which normally have some kind of negative historical background or connotation.

The same phrase in Another Country may have little or no meaning whatsoever.

Sometimes the term "monkey" is used as a term of endearment, normally associated with references to children: "You clever monkey", "You naughty monkey".

Using the same Word in context to an adult in an African country for example would be seen as hugely racist and would probably get go a good beating if not worse.