How do you help a migrating belly button piercing?

Take the jewellery out of the piercing and call it a day. The piercing will continue to migrate and will eventually drop the piercing altogether. Removing the jewellery now will allow your body to maintain the tissue and prevent the formation of scar tissue. After a few weeks you could go and talk to your local professional body piercing about having the piercing redone deeper behind the migrating piercing.