What rules govern forced removal from flights (due to overbooking)?

The relevant legislation very well could be 49 U.S. Code § 46504 - Interference with flight crew members and attendants.

An individual on an aircraft in the special aircraft jurisdiction of the United States who, by assaulting or intimidating a flight crew member or flight attendant of the aircraft, interferes with the performance of the duties

From the report we have an eyewitness saying the man got very upset when told he should leave the flight. Perhaps this was interpreted as "intimidating a flight crew member".

Edit: also the contract of carriage has this to say:

UA shall have the right to refuse to transport or shall have the right to remove from the aircraft at any point, any Passenger for the following reasons: ... Whenever refusal or removal of a Passenger may be necessary for the safety of such Passenger or other Passengers or members of the crew including, but not limited to:

Passengers whose conduct is disorderly, offensive, abusive, or violent;

Passengers who fail to comply with or interfere with the duties of the members of the flight crew, federal regulations, or security directives;

I have seen several cases with companies asking 2-3 passengers to get off the flight while they are already at their seat, ready for the take off.

Why is this happening so late and not earlier? Some planes are flying close (short and mid-haul) to the weight limit when they are full. Depending on the number of luggage and number of passengers really getting into the plane, it could end up that a plane is above the limit while if you remove 2 passengers, it is ok. This isn't something they could really have predicted in advance as the luggage is unknown until they are checked in and on the same way, some people check in online but never show up at the gate. So you only really know the total weight once passengers and luggage are in!

Now what are the criteria? First let me tell you that the one being designated by the airline will always find it unfair. And it is! If we are taking the plane on a date, it is usually because we are in some kind of hurry to get to another place. Nobody is taking the plane one week in advance :)

So for the criteria, the focus will first go on people that aren't on a transit flight as the company doesn't want to delay a larger travel. Then, if they need to disembark 2 people, they will check if there are 2 persons traveling together willing to get off the plane (this reduces the potential hotel costs for the airline). They certainly avoid hitting loyal or higher classes travelers. While nobody will explicitly confirm this rule, I have never seen a first class passenger being forced to get off the plane. I am sure there are other written and non written rules but I am not aware of all of these.

And I am not surprised that they call to call some policemen to make it happen. Is it shocking? Certainly! But surprising not really. People being asked to disembark can have all kind of reactions from crying to shouting or even hitting the crew. The police will eventually in this situation...

I knew that airlines could overbook flights and deny boarding to some passengers, but I thought that kind of boarding denial would happen at the gate and that once folks had been let on, those denied boarding would be those arriving later to the gate.

In the case of the United 3411 incident, this happened because the flight needed the seats for staff members who had to cover an unstaffed flight in a "downline connection".

From Wikipedia: United Express Flight 34111 incident

After passengers were seated in the aircraft, but while the plane was still at the gate, the Republic Airlines flight crew announced that they needed to remove four passengers to accommodate four staff members who had to cover an unstaffed flight at another location. Passengers were initially offered US$400 in vouchers, a hotel stay, and a seat on a plane leaving more than 21 hours later if they voluntarily deplaned. With no volunteers, the offer was increased to $800.

From USA Today, quoting United

“They were considered ‘must-ride’ passengers,” he said. "It was all about repositioning the crew."

Guerin acknowledged the United initially categorized the flight as overbooked as news of the video grew, but is offering the “clarification” now that the company knows more facts about the incident.

So, the answer to your question seems to be that passengers can be removed from the seats if the airline decides that a "must-ride" passenger (such as crew needed for an unstaffed flight) need them.