Can I use a priority seat if it is vacant?

As someone with one of those "hidden disabilities" that make me unable to stand for prolonged periods at times (and especially in a moving bus or train), I'd say indeed use the seat if there's no other available but be prepared to give it up to someone who needs it more than you.

It's not nice to have to ask someone to please give them a seat (more because I'm loathe to ask for help than because I don't want to be seen as weak or something like that), but it's better than being prevented from getting to a seat at all because someone's blocking the aisle you need to move through to get to it :)

For me it's luckily only an occasional issue, as my problems are related to recurring but not constant arthritis attacks and back pain that severely affect my balance. Mostly I don't need crutches but sometimes I do. And sadly, though not common, there are cases where other passengers will seemingly deliberately hinder those in need. Compared to that, just a minor inconvenience of waiting for someone to lend you a hand and give you a seat is nothing.

In general if there is a seat available, you should aim to remain seated while the train, bus or tram is moving. This is a simple safety consideration: it's more dangerous to be standing than seated in the event of a sudden stop or crash. This includes utilising any seats designated as priority.

Priority seating labelling varies across networks, with the more modern signage saying something to the effect of

Priority Seating. Please be prepared to give up your seat to passengers with mobility needs

I think the key word here is prepared, you should be on the look out for others that might need it more at stops or stations if you are seated in a priority seat, not glued to a phone, asleep or otherwise oblivious to your surroundings. Obviously if you're sat in a normal seat and someone asks for it you should of course oblige, that's just common courtesy, but the be prepared part is less relevant.

Transport for London says the following:

All buses, Tubes, trains and trams have clearly marked priority seats for anyone who needs them. If one isn't available, ask if someone will give up a seat.

Source: Transport for London website

The implication here is that they can be used by anyone but they should move if asked.

CrossCountry trains is more explicit:

Other passengers may sit in these seats, but wherever possible they should be offered to anyone who needs them as a priority.

Source: CrossCountry Trains website

West Midlands Trains also indicates the able bodied may use priority seats:

When asked politely, most people will be quite happy to move from the seat (unless they need it too) to allow you to sit there instead.

Source: West Midlands Trains website

Other networks I looked into were less explicit in their language, merely referencing priority seating, but equally none of them state able bodied passengers making use of those seats is prohibited. Virgin Trains (which typically runs longer, inter-city routes) states that priority seating must be reserved in advance (Virgin Trains).

Priority seating is not to be confused with the blue badge disabled parking system where it is a criminal offence to park in (publicly owned) disabled spaces without displaying a valid blue badge.

You can sit on them if no-one else is sitting in them. Be ready to offer your seat though.

If there are other available seats you should sit there instead as some people may have hidden disabilities that you don’t know about. You also won’t have the inconvenience of having to change seats later.

The person your offering it to might resent the implication that they need it. They may be offended that you think they look old or weak. Even more embarrassing, you might make the assumption they are pregnant, when they just look it.

In conclusion you can sit there but if there are other seats I wouldn’t.

Just like with "disabled" bathrooms, anyone can use these seats up until a disabled person requires them. There's no point in avoiding them just because someone might be too shy to ask for a seat. All you need to do is give up your seat at first notice - either when you see a disabled person or when someone (visibly disabled or not) asks you to vacate it.

Same rules apply to regular seats too - if someone with a cane needs a seat, I'd give it up even if there are special seats somewhere else on the bus/train.

Yes, you can use them. This is the fundamental difference between Priority and Reserved seating. On most types of transport, you have Priority seating which means that a person that fits the criteria (elderly, pregnant, etc) gets the seat in priority to one that does not.

As long as it is available, you can take it but I also suggest that you take a non-priority seat if there is one available before using a priority one, although most people seem not to follow this suggestion. The main reason is that if you take the priority seating, someone who would need it more might not request it and make quite an effort to get to a seat which is available. I see this happen often but I also realize that priority seats are often over-provisioned.

In some places you have Reserved seating and the wording implies that you cannot use those seats even if they are available.

As there is some disagreement, lets go with some sources as well:

Priority seats have been designated in public transport vehicles by certain transport operators to allow elderly, disabled, pregnant women and the injured to ride public transport with an equal degree of access and comfort as other people. Priority seats can be found on various public transportation, including the mass transit railways, buses, minibuses, and trams. The slogan "Please offer your seat to anyone in need" is often displayed beside the seat. The elderly, disabled, pregnant, and injured have priority to take these seats. In most cases, there is no regulation to restrict the use of priority seats, but people are expected to offer their seats to those in need.

[...]

[..] people think that only people in need can sit on the priority seats. Even if the train is full, priority seats are still left empty. This situation is common in Taiwan and Hong Kong. People, especially young people, are not willing to sit on the train as they are afraid of being morally criticized, scolded or even cyberbullied (like uploading their scenes of occupying priority seats to social networking websites). However, the priority seats are first-come-first-served. Priority seats are designed to promote the culture of offering seats to the needy. If there are no needy on the public transport, people are free to sit on the priority seats.

Source: Wikipedia

So, practically the take away from a travel perspective it might be wise to not use the priority seating if you're in Taiwan or Hong Kong, because even if you have the right to use them as an able-bodied person, it can still cause negative reactions.

Beyond that though in general you're allowed to use them, although at least I personally try to always sit down in a non-priority seat if possible. Both because that means I won't have to move out of the priority seat if someone needing it shows up and because it makes the lives of disabled and elderly people far easier.

You should be aware that some cities (personally familiar with Toronto but it sounds like London has something similar) have programs with official badges for people with invisible disabilities, which look like this (program details):

If you're in a city with such a program, you should look out for these and move from a priority seat if you see someone wearing one.

Should I continue to avoid using priority seats even if they are vacant?

Yes, unless you're really sure no-one else needs it (e.g. multiple priority seats are available and you're the only one standing) or it’s so crowded that you don’t really have a choice.

Keep in mind that it is not always obvious when people need a seat and they often not dare to ask, priority seats should as far as possible be left empty for people needing them.

Related answer: https://travel.stackexchange.com/a/129527/55572