Can a president be removed from office because he is too drunk to speak on the day of him being sworn in?

No, that would be unconstitutional.

The constitution itself does not handle such an instance which would be very unlikely to happen.