What should I pay for a 5-km songthaew trip in Koh Samui?

The regular public Songthaew buses have fixed price and fixed route. One can not negotiate the price. It is 10 Baht in rural towns up to 20 Baht (and sometimes even 30) in crowded areas full of tourists (like Pattaya, Hua Hin, or on the islands).

Once you order where to go, it instantly turns a private charter, and you negotiate the price. This holds true even if the rest of the passengers go by the standard route¹.

The OP asks for a fair price. The fair price depends on the route: going 5 km along the highway (e.g., Samui Aquarium to Lamai beach) would cost 100 Baht, while going to a less populated area can be 250 Baht or higher. Going to/from the airport is even higher. I take a (car) taxi in such cases.

The price also depends on the number of passengers²: Songthaew drivers tend to rise the price if many of you are going.

I would pay 200 or 250 for going Lamai to Chaweng³ if it were a private charter.

¹) Caveat: If you only talk about the direction, the Songthaew turns a taxi even despite the other passengers pay the regular 20 Baht. My mother reminded me an unpleasant story that happened when she visited me.

She took a Songthaew that was going by its standard route, with other passengers in the wagon. Instead of just entering the wagon, she asked the driver if the bus goes to a certain location she was going. The driver said yes, opened the passenger's door in the cabin (there are two passenger seats there), let her in, turned the air conditioner on and happily talked to her all the way.
When she arrived, she attempted to give him a standard 20 Baht and he was very furious on her, demanding 200 Baht or so.

²) Caveat: Always explicitly discuss whether you pay per person (/khon la/) or all together (/duei kan/). Here's a very typical dialog (in Thaiglish):

You: "Go Chaweng Beach, how much?"
Driver: "/khon la nueng roi/" (per person one hundred)
You: (omitting /khon la/ because you don't know the word) "One hundred?"
Driver: "Yes, yes, one hunlit."

After this dialog, you are convinced it is 100, and he thinks it is 200 (if there are two of you). After the arrival, you will have an unpleasant surprise and unnecessary dispute.

³) It is not 5 km. Lamai to Chaweng Noi is 7 km. If you meant the "Big" Chaweng (near Bo Put), it is 12 km — even if you stop straight on the road and never turn into small sois.