What does a person mean when they say that is not a compliment?

This is a condescending way to phrase an insult, and may itself be considered impolite.

The speaker is expressing criticism of a certain behavior, and suggesting that the person being criticized is not even aware that the behavior is considered bad or wrong.

An example would be to call someone "loud," which could be a statement of fact, or praise of their speaking volume. But it usually means "too loud."