How were the denominations of US coins selected?

Mostly by simple need for all the more popular coins. The cent and half-cent were needed to provide the smallest part of our coinage system. Then the half-dime and dime were needed to provide basic coins. The quarter and half dollar were used for larger denominations of coins and were easily divisible. And then the dollar coin was first minted for larger sums, but mainly was exported to the Caribbean where it was exchanged for 8 real (slightly heavier but traded on a 1:1 ratio with the US dollar), the 8 real would then be melted down to the specifications of a US dollar, recoined and the people who imported/exported these coins would keep the profit. For US gold coinage, they were used for larger amounts of money, such as the $2.50 quarter eagle, the $5 half-eagle, the $10 eagle and the $20 double eagle. Because the founders of the US distrusted paper money, gold was generally the way that the wealthy would store their wealth, along with providing easy ways of transferring sums of money between banks.

The less popular and more obscure coins though have different reasons.

The three-cent piece was made mostly because of the change of postage from 5 cents to 3 cents. The gold three-dollar piece was made for a similar reason, to easily buy sheets of stamps. The $4 gold Stella (mostly a pattern coin) was minted so the US could join the Latin Monetary Union, it was meant to circulate basically at par with the 20 Franc gold coin. The copper two-cent piece was designed in times of low inflation and its worth back when it was minted was about the same value as a modern-day quarter which would naturally be useful. The ill-fated twenty-cent piece was made to help US exports and international trade with it being circulated with about the same silver content as the silver Franc which was used as a reserve currency for nations to store their wealth in Europe. And as for the modern nickel 5 cent piece, it was basically created in response to the nickel lobby wanting congress to help them out, so congress minted the nickel and eventually retired the old half-dime.