How do you 'self-isolate' in a hostel dorm room?

Go home if you can!

If you are travelling in your country of residence, go home and self-isolate there. If travel restrictions make this impossible, contact the relevant authorities for advise. Usually there are some exemptions for essential travel, and travelling from a potentially crowded hostel to your home may well count as valid travel.

If you are travelling in another country than your country of residence, contact your embassy. Many embassies are attempting to repatriate their citizens. Chances are your embassy is already working on trying to reach you if they know how to contact you (I received an email from my government already, but I have my home and job here and the situation in my mother country is no better than here, so it makes no sense for me to repatriate).

Now is not the time to travel for non-essential purposes.

In NZ, (and I imagine Australia) you can't

Police removed a traveler from a hostel yesterday and have forcibly quarantined her and she will deported at the end of the 14 days. There are now three backpackers currently in custody for not following self isolation correctly.

At this point you need to have a correct self isolation plan when you enter the country. If the Ministry of Health doesn't consider it appropriate or think the traveler is taking it seriously, they won't be allowed to reach customs.

traveler who is planning on staying in hostel dorms

Make a different plan. Your question assumes that the hosteler simply will not change accommodation plans and will perform the self-isolation in a hostel. Not gonna happen.

The hosteler must find out what the government considers to be a viable self-quarantine plan. And then make that plan, whatever it takes. Get to a different space where self-isolation is actually viable. Upgrade to a hotel room, find a friend willing to accommodate, secure food and entertainment supply etc. per what the government considers reasonable.

And they better be able to answer questions about that plan if immigration asks.

If the hosteler is unable to change plans, then it's time to either self-arrange to leave the country... or contact the embassy and ask for assistance in repatriation... or avail local government assistance. That rubs wrong a visitor's agreement not to rely on government services, but I'm sure the government would prefer you ask for help in this case.

If the hosteler is unwilling to change plans, that misses the entire point of immigration, which is that you agree to be a lawful citizen and a boon to the nation (e.g. bringing tourist activity and funds). Flouting the law deserves the swift boot.

As Dave Smylie reports, that is exactly what is happening in New Zealand to a pair, and to a single. In one case, the fundamental flaw was a 2 week visit (that would be spent entirely in self-isolation, then fly home). There were no travel adjustments, change to an appropriate facility (e.g. hotel with viable food service, microwave etc.), no stocking on food or entertainment. The only thing that made sense was that the tourist intended to flout the restrictions. So the boot they got!

Worse, here's the mechanism: They are in "jail", immigration detention, where they will sit out the 14 day isolation. After that, they'll be given a deportation order, and freed on their own recognizance to arrange transport, gather their things and self-deport. I call that "giving them enough rope to hang themselves" :) Of course Immigration is watching, and their conduct will decide whether they ever again visit NZ, or countries that NZ interchanges data with.

After more than a year of COVID rules, many hostels which are still open offer only 'single occupancy' rooms. That is, one party per room, with a maximum per room.

If you are traveling alone, you might be able to rent a two bed 'dorm' or an eight bed dorm, but you will be the only user in it. (And expect to pay for all beds in that room, so way more expensive than usual hostel prices.
This only works if the room or dorm has private use bathrooms and in those where bathrooms are en-suite.

In those hostels you might be able to self isolate, but only if the hostel does follow all those rules and it might well be that the rules in the country you are in do not allow self isolation in hostels or in other 'not appointed' places to stay.

There are instructions for people who are self isolating in a shared space. They are aimed at people sharing a spacious home with their family, but they can apply here too. There are summaries by the trusted news sources CBC and BBC that quote medical authorities.

You should:

• stay inside. Don't go outside, especially to restaurants or grocery stores. Arrange for food and supplies to be brought to you
• if you live with others, stay 2 metres away from them and do not touch them. Do not share a bed. Do not share dishes and cutlery even if you wash them well between uses.
• use a different bathroom if possible. The bathroom you use should be thoroughly cleaned every day with ordinary household cleaners
• if possible, have others bring you food on disposable dishes and keep the garbage in your room until the isolation is over. (A friend doing this keeps metal cutlery in her room and washes it in the bathroom.) If that is not possible, only use the kitchen when no-one else is there, and clean thoroughly (again, with ordinary household cleaners) just before you leave.

The difference in a hostel is that the people you live with may not care about you as much as housemates or family do, and that there may not be separate bedrooms for each person, much less separate bathrooms. Nonetheless, get as much separation as you can and use separation-in-time when you can't use separation-in-space. Use your phone to organize and order things, pay for things, etc.

With current occupancy rates, I'd expect you to have high likelihood of being alone in the dorm room and thus being able to self-isolate. OTOH it's completely different question how authorities will like this idea, will the hostel be still open when you get there, will immigration let you into the country, etc.