Who pays for the return air ticket when a country refuses entry (by air)?

It is sometimes possible to stay in the airport long enough to catch a flight to somewhere else, depending on the view of the immigration officer. Obviously, in such a case, you would pay for the ticket yourself, and I imagine your airline would support your petition. But if you mean a simple "No passport/visa? You're not getting past this desk.", then the conventions state that it is your carrier's responsibility to return you to whence you came.

(I have always understood that this is the reason why you have to present your passport at the check-in desk. No country requires a passport to leave the country, but the airline wants to know that you will be admitted before they allow you on board.)

EDIT: The Warsaw Convention specified that in cases where a passenger is denied entry, it becomes the responsibility of the carrier to transport the passenger back to the starting pouint. It did not, so far as I know, specify whether a charge could be made for this. However, the point seems to have been overtaken by events; immigration law in most countries (the United States and United Kingdom, at least, have published theirs online) specifies that if a passenger (sea or air) is denied entry, the carrier is liable not only for the fare but for a large fine as well, unless the passenger has misled them. In practice this means you will be transported back, and the airline will try to prove everything was your fault. Now is the time you find out what sort of legal advice your travel insurance provides.

then who pays for the return ticket?

The IATA Ticketing Handbook sets out the procedure for inadmissible passengers in Section 2.23.13.2.

The Handbook is in copyright so I shall paraphrase, however it is not too difficult to find online if you want to read the original wording.

The final inbound airline is responsible for ticketing the inadmissible passenger to whatever place advised by the authorities. It should obtain payment in the following order:

1. If the passenger already has an outbound ticket (on any carrier) to the place specified by the authorities, then that ticket may be used immediately. Any restrictions preventing immediate use of the ticket (such as minimum stay, change fees, restriction to a specific airline, etc) should be ignored. The ticket should be revalidated (changed) by the final inbound airline and imprinted with the words "RESTRICTIONS WAIVED DUE TO INAD" in the endorsements box (to explain the situation to other carriers).

2. If there are unflown flight coupons remaining on the inbound ticket, the final inbound airline may appropriate these flight coupons and use their value towards payment of a new outbound ticket. (Again, the ticketing airline may disregard any endorsements or restrictions on those coupons.) The word "INAD" must be appended to the passenger's name on the new ticket.

3. If the passenger does not have a return ticket or the value of the unflown coupons does not cover the transportation costs, then the final inbound carrier is responsible for collecting the fare due for the outbound journey from the passenger.

4. If the final inbound carrier is unable to collect the fare due for the outbound journey from the passenger, that carrier is still responsible for issuing the outbound ticket. The cost of the outbound ticket is shared between all the carriers who offered carriage on the continuous journey from the point of origin on the ticket (or the last stopover, if there was one), to the place where entry was refused. The sharing is pro-rated by mileage, so the final carrier won't pay much if it just operated a short leg on a much longer itinerary.

5. However, if the final inbound carrier was technically incapable of issuing the onward ticket, then another inbound carrier may issue it instead; but the final inbound carrier is then responsible to the ticketing carrier for the full transportation cost (it isn't shared).

ICAO Annex 9 chapter 5 covers removal. Entries 5.10 and 5.11:

5.10 When a person is found inadmissible and is returned to the aircraft operator for transport away from the territory of the State, the aircraft operator shall not be precluded from recovering from such person any transportation costs involved in his removal. 5.11 The aircraft operator shall remove the inadmissible person to: a) the point where he commenced his journey; or b) to any place where he is admissible.

PDF download

If you are refused entry and came by air, the airline that brought you there has to take you back. Depending on the local law, it might also be fined if you did not have a visa and it failed to check. The airline might try to recover the funds from you later but nobody is paying for a ticket.

If you are refused entry on a land border, nobody will pay for a ticket either, you are just unable to enter the country and left stranded wherever you are. In some cases, when the control does not happen at the border itself but further inland (either on board a train or at a station), I have seen people forced to take a train in the other direction but I don't know what the rules were.

If for some reason you can't return where you came from (say you don't have the right to reenter the country you just left), the only thing left is to deport you somewhere else. Reasonable countries will at least try to deport you to a country from which you are a citizen. In that case, the country that wants to deport you pays for your ticket (and if necessary those of your police escort). Again, it might also impose a fine and try to recover money from you later but I suspect that most countries don't bother as many people who are deported will have very little money to begin with or come from countries where effective means to recover a fine are non-existent.

So in a nutshell, the country that sent you away or the airline that transported you might try to recover some money from you later on but it really does not matter if you have any at the moment you are denied entry. Either the airline will take care of the transport directly or it will be paid by the state removing you.