Airport security checkpoint controlling carry-on luggage amount?

In addition to notes and posters near the check-in counters, I have seen three different styles of checking hand-luggage policy:

1. Airport security during security check (most airports)
2. An airport employee before the security check, sometimes with a scale (for example, CMN)
3. An employee hanging around the check-in counters, checking passengers and advice them (for example, BOM)

These patterns can change from airline to airline and from airport to airport, but usually the security personnel are the last resort and they do check this. Also if the luggage does not fit in the x-ray scanner then it is a no-go. From the TSA's official page:

Carry-on baggage is a small piece of luggage you keep with you in the cabin of the aircraft while flying. You are allowed one carry-on in addition to one personal item such as a laptop computer, purse, small backpack, briefcase, or camera case.

TSA will screen any "Carry-on" baggage that will fit through the x-ray machine; however, it is up to each individual air carrier as to whether the baggage fits the size restrictions for your flight. Please check with the air carrier prior to proceeding through the security checkpoints.

Remember, at the end it is your responsibility to make sure of the hand-luggage policy, if you manage to get to the plane with luggage does not fit the policy, they will simply offload it and send it to the cargo.

In the USA, the airline will accept purchases made after the security checkpoint because of the liquids rule. You can have liquids 3oz and under in a quart-sized bag at the checkpoint, and purchase larger bottles of liquid (eg, bottled water) after.

Security does not enforce an airline's carry-on policy -- that's not their job.

At some airports or some terminals of some airports (all the example I have in mind are in Europe, it did not happen to me elsewhere), the security check happens directly at the gate. In that case, the personnel might know about the airline's policy and check accordingly.

In one particular case at a small German airport, I remember being asked to go back to the check-in desk to check in a bag because it was too heavy in the opinion of the person manning the x-ray machine (this person was working for some private security firm, neither a governmental agency like the TSA nor the airline or a handling agent). I am not saying that it's the case everywhere or that it's what the rules prescribe but this is one way it can happen.

There are airports where the airport security staff do enforce rules regarding the number, size, or weight of bags. This has become less rigorous the past few years in the U.S. and U.K., but does still happen. Often, the rules are put in place with an explanation that it limits the amount of work the security screeners need to do and hence speeds up the lines. In particular, LHR security screeners sometimes get very picky about enforcing the two bags per person rule, and there have been times when they were enforcing size limits as well (but they haven't been doing much of that lately). Hong Kong has a checker just prior to entering security who eyeballs your bags, and if they are too big, won't let you through unless the airline check-in desk added a tag indicating it has been approved for carry-on (in which case the checker tears off part of the tag). U.S. security screeners often enforce the two bags per person rule, but purchased food doesn't count.

To answer your question of what would happen if you tried to go through security with five plastic bags, the short answer is that it depends on the policy of the airport and the mood of the specific screener. If the bags are small enough, the checker is likely to let you through. Some checkers will insist that you consolidate them into two bags.