What is the proper etiquette at Japanese onsen?

In addition to codinghands's great answer I would like to add a few pointers:

1. Make sure the onsen provides a large and small towel for visitors or bring your own
2. While you can't wear any clothing in the bath, you can cover up with a small towel
3. When in Rome, do as the Romans

Towels

Depending on the onsen, towels (a smaller washcloth, and a larger towel for drying off) may be provided free, provided for a fee, or not provided at all.

If you are unsure which category the one you will visit falls under, bring a small towel (a washcloth essentially, something around 20 x 20 cm), and a large bath towel to dry off with afterwards. Some onsen may also not provide soap and shampoo, and so you may want to bring those along as well.

Covering Up

As explained by codinghands, onsen rarely allow swimsuits or any other sort of clothing. Onsen are regularly featured on TV with people wearing large white towels when they go in, but in almost every case there is a disclaimer saying, "They are allowed to wear towels due to special permission from the onsen" and this is not something you should copy.

While you can take precautions to cover up (as explained below), the reality is that people are going to see you unclothed and no amount of trying to prevent that from happening is going to make it work. You are taking a bath with people, and this means that they will see you without clothes on. If this makes you too uncomfortable, then don't go to a hot spring.

Many people (Japanese included) use the smaller washcloth-sized towel to cover up when walking around by holding it in front of whichever parts they would like to cover up. In the case that you use the washcloth after having washed (with soap) with it, be sure to rinse it out thoroughly as you want to avoid getting soap in the actual bath.

When in Rome

codinghands created a nice handy guide on how things (usually) work. But there are quite a few steps, and there are always going to be variations depending on where you go and who's there. Do at least as much as the other people are doing and you should be fine, though feel free to do more. The basic concepts are:

1. Be clean so you don't get the bath dirty
2. Don't get anything but yourself in the bath (no soap, towels, etc.)

That said, there's a chance you'll see people come in, shower with water (without soap or shampoo first), and hop in the bath. If everyone else is doing that, feel free to do the same (you can soap up and shampoo first if you'd like, of course). But if everyone is soaping and shampooing thoroughly, follow their cue. In the same way, you will see people hop in the bath with their towel covering themselves. If everyone is doing this, feel free, but if nobody is then make sure the towel doesn't go in the water.

A few years ago I wrote up a comprehensive guide on going to and getting in onsens for my local website (I love onsens, and have lived in Japan for 8 years). The above answer is correct, however if you'd like more details as well as a start-to-finish getting-in-the-onsen guide, feel free to check out the "Onsen Experience" section at the following link. Onsen - The Mie Guidebook