A better rule of thumb is to not exchange any cash at all, but use an ATM in the country of arrival to withdraw money, letting your home bank do the currency conversion (so withdraw in HKD, not in EUR; the ATM itself will offer Direct Currency Conversion but that's usually a much worse rate for you). It's also quite logical; HKD banknotes in the Euro zone have to be transported there, so they are more expensive.

That leaves the problem that you might have some HKD left when you leave the country; as long as it's not too much, I tend to keep it for the next visit. Or you can buy something (souvenir, drinks) at the airport. (I don't have any experience in whether it would be better to exchange it back to EUR in Hong Kong or in Europe, so I'll leave that part to another user.)

Anecdotal evidence: in some (Western/Northern European) countries you don't even need cash at all. I recently visited Iceland and the only moment when I didn't have the option to pay by card was the offerings during the church service...

I concur with Glorfindel, the best option is usually just to withdraw at local ATMs and use your card directly for most payments.

A few caveats, though:

• You must of course have a card that can be used internationally and widely accepted (should be the case of all VISA, MasterCard and Maestro cards).
• Check your card issuer's fees for withdrawals, payments and the exchange rate they use. It can vary quite substantially. Note that in most cases, it's cheaper to pay rather than withdraw. Some banks may have an international presence or international partners where fees are lower or waived.
• Make sure you inform your bank that you are travelling abroad. Some have dedicated interfaces for that on their online banking service (you specify the country and travel dates), others will require you call them.
• Check that the ATM you are planning to use does indeed take the right type of card. There are stories of cards "swallowed" because they're the wrong type.
• Take a bit of your own local money to exchange just in case there's a problem.
• Usually, refuse the offer for currency conversion by the ATM or payment terminals, though of course you should compare with the rate your bank offers.
• In some countries, some of the ATMs may add their own fee when withdrawing. This is becoming quite common in the US.
• Make a note of your card details and emergency contact before you leave.

In most cases, fees (by your bank or by the ATM, other than currency exchange) are usually a fixed amount per transaction, so larger withdrawals/payments cost you less, but YMMV.

In my experience, if I have to change currency (and cannot withdraw cash as the other answers suggest), usually the rates seemed better at the country of departure, probably because you can choose to do it outside of airports, so more choice means better rates.

However, the one exception I've encountered was Singapore: exchanging Euros there seemed much, much better than buying Singapore dollars in Europe. But Singapore is a hub country full of traveling foreigners, which probably explains it.

If Hong Kong follows a similar pattern, then it could be better to sell euros there, instead of buying HKD in Europe.

Not a big deal but

Just one point nobody mentioned...

Say you're in Local country and you have (exactly) 500 units (5x 100 notes).

When you go to Distant Country, you can give them the 500 and get in return in your hand

the precise amount of local currency, to the cent (peso .. yen . whatever) including small change.

If you change in Local country, they don't keep small change in Distant Currency. You get annoying rounding issues.

In general for this reason, and the rate, you always change "there" not "here".

(If, for some reason, you archaically are dealing in cash.)