Can a flight refuse to check in bags of correct size if there are too many?

Yes, this can happen. The fine print on airline websites usually have rules on this. To pick one example airline, United says:

Please note that extra baggage is accepted only if space is available on the aircraft. Oversized, overweight and extra baggage is not accepted on flights to certain destinations and on certain flights during specific timeframes. For a list of cities where overweight or extra baggage is not accepted, either year-round or during certain times, please see the Excess Baggage Embargo page.

The baggage embargo page then lists further restrictions. And their Contract of Carriage says:

UA’s acceptance of Excess and Oversize/Overweight Baggage shall be on a space-available basis only, and shall be subject to the load capacities of the aircraft in use. United may prohibit Checked Baggage exceeding either 70 lbs or more than 115 inches.

As noted in other answers, it's also possible, especially with smaller planes, for your baggage to be accepted but for it to have to take a later flight due to capacity constraints.

Yes. This happened to me last year. They allowed the baggage allowance of two suitcases which are included with the price of the ticket yet I had one more, of the same size and weight and they refused it even though the airline website indicated that one can buy allowance for up to 8 more. I specified that I would pay as indicated and did not mind if it went on the next flight it was still refused. The additional suitcase had to be sent by courier which was a stressful and complicated thing to do on they day of the flight. The really awful aspect of this is that they did it on the return flight but not the ongoing one.

This was a Delta flight and their Extra Baggage Terms clearly say this - although I had missed that information apparently:

We may limit the amount of excess baggage when such excess prevents us from transporting up to two bags per passenger.

And:

Any bags exceeding the ticketed itinerary allocation: number of pieces, weight or size are accepted on a standby basis.

This will depend on the terms of carriage for the particular airline, and the consumer law of the country from which the flight is departing.

In general, an airline won't decline to accept baggage, but there is no guarantee that it will travel on your particular flight. It may be delayed and show up on a later flight, perhaps later that day or even the next day or after, depending on demand for cargo space on flights on that route.

Additionally, it's going to be very aircraft-dependent. Some aircraft have a lot more luggage capacity than do others. You're far more likely to talk an airline into letting you take five bags on a Boeing 767-300ER than you are on a Canadair CRJ-200. If an airline usually has extra space for baggage on the route, it will be more willing to take your excess bags, because it has some hope of delivering them to you on a timely basis.

It is unlikely that the airline will communicate with you about the odds of this happening, for security reasons. They prefer that passengers believe that they will be traveling with their baggage.

Remember, too, that if you want to ensure your excess baggage arrives somewhere reasonably promptly, you can always send it by air cargo, although this means it is unlikely to travel with you on your aircraft. (It may take another route if this gives it a better chance of a timely arrival.)