Is it bad luck to remove a gnome from your garden?

No. I removed mine and replaced it with a garden Buddha which looks much nicer. The luck of my gardening has not changed in any way.