How can you congratulate a coworker for a new baby?

If your only relationship with the co-worker is at work, a card would be sufficient. Check with the others at work to see if they are sending a "group" card &/or gift.

If the co-worker is a friend as well, a card and small gift (for the new baby) work well.

If you are close friends as well as co-workers and intend to go by to visit with her (and meet the new baby) then bring along something small and thoughtful for the baby as well as something to pamper the new Mom.