Why do I need to buy a ticket to get into the secure half of a US airport?

You have it backwards. Nobody from the airline is saying "if you want to get in the secure half, you have to buy a ticket." They are saying "the secure half is only for people who are flying somewhere today." People who want to get in the secure half even though they are not flying today don't want to be told they can't do that. (Whether they are bad guys including pickpockets and con artists, or good guys who want to accompany or meet someone, is not relevant.) Then someone realizes that in fact it's impossible to distinguish "people who are flying somewhere today" from "people who bought a cheap (or expensive but 100% refundable) ticket they don't intend to use."

There is no way this is a fee. (For one thing, the fully refundable version of the strategy costs you nothing.) The airline and the airport would both prefer that people who want to go to the secure half even though they are not flying would just abandon that want and stay out of the secure half. But people are people and have found a way around the rule.

Why do the airline, airport, and security folks have this as a rule if it's so easy to circumvent? Several reasons:

• security theatre. The signs say that only people who are travelling are allowed through here. See, we're protecting you by not allowing the public in. Don't you feel safer?
• genuine reduction. Not everyone knows the cheat. Some know but would never do it, they abide by the rule because they are rule abiders. This reduces volume of non flyers through security
• a chance for checking - to buy a ticket and to get through the first checkpoint, the name in which you buy the ticket must be checked against the no fly list and your id (at least in the US) is checked to confirm that you are actually the person named on your ticket.

Is it perfect? No. Thieves and terrorists, who have something to gain by getting into the secure half, may be willing to jump through some hoops to get there. But let me close with this: can you think of a way to tell apart those who are flying today from those who have bought a ticket for a flight today? What would you change the rule to, in order to improve it?

I don't have any specific insight into this but my guess is that designating an area of the airport to be off-limit for people who don't fly is useful to manage things like security, duty-free shopping, waiting areas, etc. Otherwise, you would either need to (re)check everyone at the gate or pay for facilities (security check, cleaning, etc.) for a larger number of people. The link posted by Kate also suggests this became a requirement after 9/11 so it does seem to have something to do with security (or the cost and delays of the more stringent security checks introduced then). It also affords the authorities an opportunity to check your name against their databases (especially the infamous no-fly list).

Note that whatever the underlying reason, the requirement is in all likelihood very effective even if it cannot be fully enforced. Buying a cheap throwaway ticket and checking-in just to visit the departure lounge might be relatively “common” but I am pretty sure that many more people prefer not to cheat, do not bother or perhaps do not even think of it.

If there was no filtering whatsoever, those people might just hop in to meet a friend in transit, grab lunch, shop or whatever and you would have larger crowds to deal with in the secure area of the airport. In particular, if you would somehow allow everyone to go through the security check without actually wanting to fly, you would need even more personnel or create even longer delays at the checkpoint.