Why do small bird dive bomb birds of prey and why don't the birds of prey retaliate?

First of all, the smaller birds are likely trying to defend a nest. Red-Tailed Hawk vs American Kestrel for example, I have seen it happen. As to why the larger birds don't fight back, I'm not entirely sure. Have you ever tried to swat a fly with a Baseball bat?