How to avoid letting a rental business hold my passport in Thailand?

The answers above are not only generic, but inaccurate for Thailand.

First and most important question is whether you are actually renting a motorcycle ("big bike") or a scooter ("motorbike").

For scooters, you can always leave a copy of your passport. It is not an issue anywhere.

But for motorcycles, you pretty much have to leave your passport. There is no place that rents "big bikes" in Thailand that does not require this. I think I've read somewhere that someone was able to get around this by leaving a huge security deposit (close to value of the bike) but this is how things operate in Thailand. Right or wrong, this is how it is.

Lastly, some areas are known to rent bikes with questionable paperwork. If you are renting ER-6 from Chiang Mai, it's no problem. But you will not be renting a completely legit Yamaha R1 from Pattaya/Phuket for 700 baht/day. (Logic prevails: R1 costs ~$30,000 usd in Thailand. There is a reason why they are renting it for $23/day.)

PS. Yes, you could get in trouble if you are stopped by police with no passport. Usually, they will be satisfied with a copy of passport and visa, or a phone call to your rental agency, or a small bribe... but technically speaking, it could be a problem.

By law, it is illegal for businesses in Thailand to hold foreigners' passports in exchange for renting them scooters, motorcycles, jet skis or anything else.
Under current legislation (Feb. 2013) any Thai business found in possession of a foreign passport will be prosecuted as technically the passport is the property of the government of the country who issued it.

In addition, it is Thai law that foreigners carry their passports with them at all times so leaving your passport with a rental agency leaves you (and them) in a situation you could both be prosecuted for.

There are plenty of motor cycle hire places in Thailand, when I say plenty, I mean hundreds and thousands of them. Don't book online, just find one when you are there, it will be cheaper and it will allow you to check out the motor bikes before hiring it.

If you are ever asked to leave your passport, simply say that you need it to exchange traveler's cheques at the bank - they would always rather make the sale than lose it over this.

Most places will accept a photocopy along with a security deposit - same with cars. Just need to say you need to do a visa run and need the passport. I live here in Thailand and have never had a problem renting anything from Honda Dreams to trucks. Just make sure you check it before you ride (as there are plenty of damage-scams about) and also pay more for the insurance (and read it - it will/should be in English).

In Chiang Mai I always go to POP to hire bikes (sometimes cars too - but there are better places for them) - there are several POP's in the city (it's like a franchise), but I always use the one next to Spotlight (gogo bar on the moat) just before Thaipae Gate (down a bit from Miguels burgers, before you get to Loi Kroh where the bars are mostly situated). They do Dreams, D-Trackers, ERN6's and Kwak8's. 800's cost a cool 1,000 per day (local prices - may be higher for tourists) - the price of a car rental. Usually deposit is 5,000 Baht (they staple it to the photocopy and the paperwork that shows previous damage etc that you sign). Pop vehicles are fully insured - they advertise the fact. And no, other than being a time-to-time customer, I am not affiliated. When my car was off the road (awaiting a new engine which took several months) I had both cars and bikes off them of varying terms - there's a daily, weekly and monthly rate.

//Edit: Just to add: On the photocopy, write clearly "For rental purposes only" and sign it - there have been rare occurrences where these are have been used for other purposes (such as getting loanshark loans)! Possibly an urban myth, but advisable anyway.

I never surrender my passport. Make a clear copy of your passport and offer that instead. If refused, point out that it is clearly written on the passport that it is illegal to surrender.
If traveling with a Thai, a Thai ID card is accepted in it's stead. I also obtain an international driving license before leaving my home country, which carries some weight as additional identification.

I've never had a problem, just be tactfully aware of your emotions when dealing with a Thai, don't get angry, NEVER lay blame on the person, as this will result in loss of face.
Place the blame on other forces.

For example: "I'm sorry, I would like to do what you ask, but I can't because my government prohibits me from doing so" not "You're wrong, I don't have to give you my passport, my government says so"