Why didn't the spell work for mermaids?

No spell will work for becoming a mermaid. It can not happen. The fact you tried should be proof to you that it doesn't work. Humans can not shape change or become a mermaid. Mermaids are not real. They are make believe and only exist in books, games, and movies.