What was the custom that the court found applicable in the paquete habana case?

It was not uncommon and,acceptable under international law, to take merchant ships as prizes if they flew the enemy flag. The rational is that it reduces the enemies ability to wage war both because of the goods said vessel might carry, the profits trade brings, etc.

The Paquete Habana was, however, a fishing vessel, and it was a custom to exempt fishing vessels from such seizure. While the captain who took them tried to argue that fisherman tend to be Excellent seamen and thus are a military asset.

This is an important case regarding international law. Custom effects what is acceptable between nations much like precedent has within a country. Customary law is the early basis for many things that are now in treaty ( recall Fluelen in Henry V saying that killing the camp followers ( "the boys in the luggage") was expressly against the rule of war).