If an airline erroneously refuses to check in a passenger on the grounds of incomplete paperwork (eg visa), is the passenger entitled to compensation?

For flights originating from or destined for the EU, the Flight Compensation Regulation (Regulation (EC) No 261/2004) provides for compulsory compensation from airlines which deny boarding. It cannot be contractually overridden.

This answer suggests that an airline may be able to rely on phrases such as "determines at its reasonable discretion" contained within its contract. I believe that is incorrect for EU cases. The important thing is whether the airline's action is justified under the provisions of the Regulation, not whether it is justified according to its contract.

Article 2(J) provides the following definition:

"denied boarding" means a refusal to carry passengers on a flight, although they have presented themselves for boarding under the conditions laid down in Article 3(2), except where there are reasonable grounds to deny them boarding, such as reasons of health, safety or security, or inadequate travel documentation;

Article 3(2) states that:

[The Regulation] shall apply on the condition that passengers: have a confirmed reservation on the flight concerned and, except in the case of cancellation referred to in Article 5, present themselves for check-in, as stipulated and at the time indicated in advance and in writing (including by electronic means) by the air carrier, the tour operator or an authorised travel agent,or, if no time is indicated, not later than 45 minutes before the published departure time.

Article 4(3):

If boarding is denied to passengers against their will, the operating air carrier shall immediately compensate them in accordance with Article 7 and assist them in accordance with Articles 8 and 9.

Note the subtle difference between e.g. British Airways "reasonably believing" there is no travel documentation and the Regulation's requirement that there be "reasonable grounds [such as] inadequate travel documentation". With the latter the court will be looking to see whether or not the travel documentations you presented were in fact adequate, as opposed to what BA reasonably believed was the case.

Note: the answers generated by this question would suggest it is probably a better fit for https://law.stackexchange.com/.

Let's dissect this a bit.

1. The passenger is responsible for having proper documentation.
2. The airline is held accountable by many countries to NOT present passengers at their border that don't meet entry requirements. Accountable = heavy fines!
3. Note that the opposite is NOT true: countries typically don't care whether passengers with right of entry are denied boarding. Example: a US citizen with an expired passport can not be denied entry to the US but no airline will let you board.
4. This creates a strong bias of the airlines to err on the side of denial and typically the contract of carriage contain a fair bit of verbiage around this and what happens if it goes wrong.
5. Airlines typically use Timatic to determine paperwork requirements. Most passengers don't have access so they need to rely on websites and other resources. As far as I know, neither source is liable for the information they provide.

Now let's look at your actual case

If an airline erroneously refuses

First you would need to determine why this is "erroneous"? Unless the airline has made a mistake and admits to it, you are looking at two conflicting sources of information. You would need a legal authority with proper standing to take a look at the conflict and determine what interpretation is correct. Timatic can be wrong or outdated. Government websites can be wrong and outdated. Happens all the time.

Overall it's less relevant whether the denial was "erroneous" but whether the airline was within their rights to deny you boarding given the paperwork you presented and the information available to them.

is the passenger entitled to compensation

Let's assume the airline has accepted that their decision was wrong. Next steps depend on what't in the terms and condition of the ticket or the contract of carriage. Most airlines would probably give you a refund or some travel credit claiming it to be "good will". Any type of compensation beyond that is highly unlikely unless the airline was grossly negligent or malicious AND they don't think they can cover it up.

To answer somewhat indirectly: someone who is in this situation is probably better off initiating a dispute with their credit card company (assuming the tickets were purchased with a credit card).

Some years ago I was denied boarding on a flight, for a reason I didn't understand at the time, and can only guess at now. Afterwards, when it came time to look into a refund, the airline said we should contact the travel agent, and (of course) the travel agent said we should contact the airline. After getting nowhere, I called the credit card company. They refunded my money immediately, said they would deal with it, and I never heard another thing about it.

(In hindsight, I believe the person at the airline counter made a mistake, caused a delay, and was then unable to check us in within the security window. But of course no one ever admitted to this, and importantly, by dealing with the credit card company directly, I never even had to know the truth, much less get someone to accept responsibility for what happened.)

I looked at the contracts of carriage of four major airlines, from four different countries: United, British Airways, Japan Airlines, and Lufthansa. Three of them allow for the airline to refuse boarding to any passenger whom the airline reasonably believes does not have valid travel documents, whether or not they are correct in this belief.

The fourth (Lufthansa) does not appear to have any such clause. The clause only appears to cover cases where the passenger does not have appropriate documents, not where the airline believes they do not have such documents.

Whether or not a belief is "reasonable" is one of those things that lawyers get paid lots of money to haggle over in court. Suffice it to say that it will depend on the particular circumstances of each case, and on the case law of the jurisdiction where action is brought.

The bolding in the quotes below is mine.

United:

UA also reserves the right to deny boarding to any Passenger whose necessary travel documents are not in good condition according to UA's reasonable belief, or which otherwise do not comply with laws of the specific country the Passenger is departing from, transiting through, or traveling to.

British Airways:

We may decide to refuse to carry you or your baggage if one or more of the following has happened or we reasonably believe may happen.

...

7a15) If you have not, or do not appear to have, valid travel documents.

7a16) If you try to enter a country for which your travel documents are not valid.

7a17) If the immigration authority for the country you are travelling to, or for a country in which you have a stopover, has told us (either orally or in writing) that it has decided not to allow you to enter that country, even if you have, or appear to have, valid travel documents.

Japan Airlines:

10. JAL may refuse Carriage of, or remove, any Passenger, and in such case his/her Baggage will be handled in the same way, if JAL determines at its reasonable discretion that: ...

(3) (a) the Passenger falls under sub-paragraph (1) (b) of paragraph (B) of Article 16,

...

16. (B) Passports and Visas

(1)

(a) A Passenger shall present to JAL all exit, entry or other necessary documents required by Applicable Laws of country concerned such as countries to be flown from, into or over, and shall permit JAL, if JAL at its reasonable discretion deems it necessary, to make and retain copies thereof; provided that, even if a Passenger presents exit, entry or other necessary documents to JAL and JAL carry the Passenger, JAL shall not be deemed to guarantee that such documents comply with Applicable Laws.

(b) JAL reserves the right to refuse Carriage of any Passenger who does not comply in any respect with any of such Applicable Laws or whose exit, entry or other necessary documents are not complete in any respect.

Lufthansa:

7.1 ... [W]e may also refuse to carry you or continue to carry you or cancel your seat reservation if ...

7.1.7. you are not in possession of a valid travel document; you wish to enter a country for which you are only entitled as a transit passenger or for which you do not possess valid immigration documents; you destroy your travel documents during the flight; or you refuse to give your travel documents to a member of the crew against a receipt when asked to do so; or