Why can't I travel onwards if my bag wasn't going to make the flight?

In general, your luggage and you need to be on the same plane because if it contains a bomb then they want you to perish. In this case, it might be you've been pulling tricks with purchasing a ticket for a notoriously delayed flight in hopes you get separated from your bag-bomb. If it's not your fault then that's apparently fine because lost / delayed / misrouted baggage happens all the time. But then you don't know so apparently that's fine. Obviously whoever created this policy never heard of suicide bombers and airport screening.

Edit: this is outlined in the IATA Recommended Procedure 1739 Passenger/Baggage Reconciliation Procedures. In US Code - Section 44901: Screening passengers and property the following is mentioned:

A bag-match program that ensures that no checked baggage is placed aboard an aircraft unless the passenger who checked the baggage is aboard the aircraft.

but this was only for "any piece of checked baggage that is not screened by an explosive detection system". I am speculating this might have stayed in place even when such a system was deployed.

This is probably the policy because of the problems delayed luggage causes at the other end, specifically with clearing US Customs. While airports/airlines usually have a protocol to clear luggage in the absence of the passenger, it is more hassle and paperwork than having the passenger take the luggage through customs themselves.

The KLM Conditions of Carriage seem to support this (my emphasis):

10.2 Checked Baggage
10.2.1 General Provisions
(d) Carrier will try, as much as reasonable possible, to arrange that Checked Baggage is carried on the same aircraft as the Passenger. Amongst others, for operating or security/safety reasons, the Checked Baggage may be carried on another flight. In this case, the Carrier will deliver the Baggage to the Passenger, unless the applicable regulations require the Passenger to be present for a customs inspection.

It is very unlikely this is the policy primarily for security purposes. On domestic flights in the US, I have on occasion had my luggage flown on an earlier flight than myself. For example, recently when I flew into Newark from the UK, I had a five or so hour layover in Newark before my connecting flight, but my luggage was put on an earlier flight to my final destination.

I suspect that KLM were actually fobbing you off. The reason you couldn't get on your flight was because the gate had already closed and they weren't willing to reopen it for you.

If a passenger's bags are already on a plane but the passenger does not board, then the bags must be removed from the flight. This is to prevent people putting bombs in baggage and then not catching the flight as happened, for example, with Pan Am flight 103. (Presumably, it is harder to recruit suicide bombers.)

However, your situation is the opposite: you would have been on the flight without your bags. This happens all the time, which is why the flight attendant mentioned it to you. Typically, a person can get between two gates at an airport faster than a bag, which must go through the central baggage handling facilities. In this situation, it is completely routine for the bag to be sent as cargo on a later flight. One assumes they put the bag through more intense screening, to prevent people putting bombs on planes by choosing a very short connection in the hope that they make the connection and their bomb blows up some other flight to the same place.