Travelling to Mexico and Central America, stop in US and no return flight booked

It would be taking it too far that Mexico is "treated in the same as the US".

But even without going that far, it does look like you have a problem. One of the criteria for being eligible to enter the US under the VWP is that you have a "round trip ticket" that takes you somewhere else -- and for this particular purpose, contiguous territory (that is, Canada and Mexico) are not considered "somewhere else" enough.

The relevant rule is 8 CFR 217.2(c)(1) which says

Applicants arriving by air and sea. Applicants must arrive on a carrier that is signatory to a Visa Waiver Pilot Program Agreement and at the time of arrival must have a round trip ticket that will transport the traveler out of the United States to any other foreign port or place as long as the trip does not terminate in contiguous territory or an adjacent island; except that the round trip ticket may transport the traveler to contiguous territory or an adjacent island, if the traveler is a resident of the country of destination.

(emphasis added). This implements the statutory requirement in 8 USC 1187(a)(8). There's even a legalese definition of "round trip ticket":

Round trip ticket means any return trip transportation ticket in the name of an arriving Visa Waiver Pilot Program applicant on a participating carrier valid for at least 1 year, electronic ticket record, airline employee passes indicating return passage, individual vouchers for return passage, group vouchers for return passage for charter flights, and military travel orders which include military dependents for return to duty stations outside the United States on U.S. military flights. A period of validity of 1 year need not be reflected on the ticket itself, provided that the carrier agrees that it will honor the return portion of the ticket at any time, as provided in Form I-775, Visa Waiver Pilot Program Agreement.

Unless the bus to Guatemala is run by a "participating carrier" (which I very much doubt) there seems to be a definite risk that you will be determined not to qualify for entry under the VWP and thus be denied entry. Exactly what that would entail is not completely clear; there seems to be a chance that you'd just be escorted to the flight to Mexico you have a ticket for anyway, but it also appears to be a possibility that you'll be put on a flight back to Europe instead.

What's even worse is that the airline might not even allow you to board the flight to the US based on an ESTA, since they know the rules too, and according to their ticketing system what you have is definitely not the kind of "round trip ticket" required by 8 CFR 217.2(c)(1).

So unless you enjoy gambling you should try to secure an ordinary US visa before you travel.