Why would passengers on a diverted flight not be allowed into the terminal?

If the airline isn't operating in this airport, then it means that they aren't paying any fee for using the airport services. Airport services aren't really pay as you use them. So a flight can't come and leave passengers like this...

Then it is also a matter of responsibility. What if the passengers start looking around and don't come back for the flight. The flight could be delayed even more and this woulf cost a fortune to the airline without elaborating on the schedule problems it could cause to the airport and others airlines.

Then for the customs, even if this is an internal flight, I guess they could do some random checks on the passenger list so if an unexpected flight comes, it could cause trouble to their organization even if I agree that it would be less of a problem than for an international flight.

My first official answer, so bear with me.

There are several reasons for this. There is a document created by the US DOT (here and more thorough one here) that pretty much covers all the reasons. It seems, primarily for the safety of the passengers (Enhancing Airline Passenger Protections). These are Tarmac Delay Contingencies.

Brief summary to answer questions:

• Since the airline had no presence at that airport, as other answers have pointed out, they have no means of providing any services for disembarkation/re-embarkation. I have personally experienced a rare case where another airline's operators helped with this and allowed passenger disembarkation. But, it's rare.
• Customs and Border Protection was involved per DOT policy. In the linked document, Customs and Border Protection is one of the agencies that should be coordinated with. The other being TSA. It depends on which one is appropriate. In the case of an airport that services international flights, Customs would probably be the appropriate one.

Hope this answers your question.

I'm sure there are various potential answers but here's a couple:

1. Some airports are "closed" with the exception of international flights. Depending on the origin country, some late arrivals are just necessary. So Customs might have had a say at that hour because it's just who was available.
2. The make-up of the passenger manifesto could play a role. For example, if you had people on your flight that required wheelchairs and those services were not available then it wouldn't be fair to them to release the passengers and have them unable to go anywhere. Same could be said for minors on-board. Much easier to keep everyone on the plane.