I would venture to say that you are loosing far more bees each season to predators that you never see, and some that you wouldn't even think of. Just yesterday I saw a Robber fly perched on my trellis, eating a baldfaced hornet that he'd caught. Life is an everyday gauntlet for the field bee.
You see all those swallows swooping down on your bees and it's right in front of you, but it probably is a drop in the bucket, overall in terms of total losses of bees. And even so, killing a few swallows, though it might make you feel like you did something about the problem, probably won't make much of a difference either, unless you've got a lot of time on your hands and you're a really good wing-shot (and lax local enforcement of the federal migratory bird act).
Like Bjorn said, it's some kind of human instinct to go out kill all of those things that we perceive as causing us trouble, or costing us money, or maybe just mildly irritating us. And you just can't convince some that it's far smarter money to take other protective measures.
If you killed everything that ever did harm to one of your bees, that'd be one big pile of dead bugs, animals, and people.