Is it a problem to use a different passport on a return flight?

Your father-in-law presumably gave his passport number to the airline when he booked his ticket. However he can use a different passport to travel on. The best way to do this is change the passport number associated with his booking online. However he can also do it at the time he checks in, or by calling the airline. None of these approaches should be a problem.

If your father-in-law is not a British citizen and requires a visa to enter Britain you need to present it to board. If it is in the old passport you need to bring the passport of course. Even if it is not bringing the passport it was associated with might be a good idea.

This is a common occurrence. It applies to renewing expired passports or stolen ones.

What works almost always is to call the airline. Some countries require passenger information to be submitted 24 hours ahead, so this is really the best option.

In most places though, it is possible to do this when checking-in in person. Online check-in usually refused to accept a different passport number the few times I had to do it. At that point, I usually call the airline to avoid issues at check-in when time can be tight.

The old passport can be brought along and used to show still valid visas but it will not be accepted as a proof of identity since it will have been voided even if its expiry was not reached.

I carry two passports, both from the same country, and use them both a lot for the same trips. I leave Hong Kong with one passport, enter and leave the destination destination country with the other one, then, when I land back in HK, use again the first passport. As long as the name and other details (gender, date and place of birth) match, there's no problem.