Is it legal for officials to hold my foreign driver's license as a guarantee for a traffic fine?

Others have already provided correct answers but it might be useful to note that the very way you framed the question is at odds with the way international law works. The relevant law in a such a situation is first and foremost the local law. Beyond some limited things like the right to request that your consulate be informed of your situation when arrested, you do not have any general right to special treatment as a foreigner, let alone immunity from local law.

Even if there were some relevant international law (which there isn't in this case, as far as I know), it is still entirely up to the local legal system to actually respect it or provide a way to have the relevant rights recognized. I am stating the obvious here but there is no “international court” where individuals could appeal national decisions, no global police to enforce international law. There are also very few general rules that could be deemed to apply to any state (customary law and general principles) and most of what matters comes from treaties and therefore only applies to selected states (those that are a party to the treaty). Even from the perspective of international law, it's therefore important to consider which countries we are talking about and you should not expect any general answer. In fact, there is a ton of international law that stems from bilateral agreements and only applies to two specific states.

In any case, treaties are routinely ignored or violated and there is usually nothing you can do about it in practice. Even for egregious abuse (think shooting a police officer from within an embassy, kidnapping foreign nationals, etc.), the main things states can do, short of starting a war, is complain officially (there are traditional ways to complain varying in the degree of severity from sending a letter to breaking off diplomatic relations and expelling the other state's diplomatic staff). And even that would still not give you the right to drive in Mexico as far as the Mexican authorities are concerned!

Also consider this: Foreigners in your country can be arrested and punished in any way that is locally acceptable (including things that most countries object to like the death penalty) without asking their home state for permission. People trying to cross a border (no matter which one) can also be subjected to a lot of arbitrariness, interrogation and detention and still don't have any practical recourse. Also, war crimes, crimes against humanity and even genocide do happen and are still very difficult to prosecute or punish in any way. Why would you expect international law to protect you effectively from a relatively innocuous measure like the confiscation of a driver's license?

Finally, you have to look at it from the perspective of the state you are visiting. It is already doing you something of a favor by letting you drive with your US license. There are in fact treaties about that and a number of states that do recognize it (with or without additional documentation) but there is no a priori reason for your US license to be recognized by each and every state in the world. So here again, no general rule that would apply everywhere.

In this particular case, Mexico has good relations with the US so it's possible to imagine the local authorities would want to avoid displeasing visitors or perhaps that there is some special bilateral agreement between the two countries but what about Cuba, Iran or North Korea? It would clearly be inconceivable for the application of US law in the US to be contingent on these countries' legal systems and vice versa. At the end of the day, what matters is therefore the local law (or whether it is respected at all by the local police, which might matter more than the law itself in some countries) and the way traffic offenses are punished does not depend on anything else.

In light of all this, what you should be asking about is probably not about what happens to US citizens in unspecified foreign countries but what happens to locals and foreigners in Mexico.

Passports and drivers licenses are governmental documents of which you are the holder. Since the Mexican police are the official representatives of the authorities in Mexico your drivers license actually is their property, so legally they have all rights to take your license.

When they confiscate your license they say that you are not fit to continue driving. Whether or not they are allowed to confiscate your drivers license for not paying a penalty depends more on local laws then on international law. If it is common practice in Mexico to take away licenses until fees are paid, then they can do the same with your (foreign) license.

If you find a loophole in the law which allows you to have two licenses, you might run into serious issues, since it can be considered a criminal offense if you continue driving on a second drivers license if your first is confiscated. When your license is confiscated abroad you first need to deal with that before driving again.

IANAL, but when you're on Mexican soil, you're legally subject to their authority. Therefore, they can legally confiscate your driver's license, and I know of no international treaty that prohibits that action. However, they can only prohibit you from driving in Mexico. I think only the US state DMV that issued the license can suspend or revoke your driving privileges in the US. I doubt the DMV will do that, because you were cited in Mexico by Mexican police.

Just to add to the examples of it's happening, there's one case that aired on TV: The TV show, The Amazing Race had the teams in IIRC New Zeland and one was driving the speed limit in mph when the signs were in kph. That was far enough over the limit that the cop took his license on the spot. They were only allowed to continue with the teammate doing the driving.

I believe that the argument that they should not be able to hold it is that a driver's license is not your personal property -- it is the property of the state or country that issued it (just like a passport is the property of the country that issued it), and that government agency can demand it back at any time. So the official who is holding your passport is confiscating someone else's property, without due process. However, I am not sure if they really care; it's not like your state is going to sue them in another country.

One interesting hypothetical question would be, what would happen if the state that issued your license were to formally demand it back from this official. What would he do?

In any case, I am pretty sure it would be legal for you to just apply for a replacement driver's license from your state. So it is not a very effective "guarantee".