TAP Portugal LHR to BSB forced shorter return date and had to pay for new return

Am I entitled to some compensation?

Unlikely.

Here is how it works:

1. When you buy the ticket, you agreed to the terms and conditions which clearly state that it is your responsibility to check and comply with all entry requirements for your destination.
2. Nothing gets checked at time of purchase: this is often many months in advance of the actual travel date. Your current visa, residency, citizenship states don't matter and your future ones are not known yet.
3. At time of check-in the airline will indeed check your current document against the requirements of destination. If that doesn't check out you will be denied boarding (and rightfully so). They can get heavily fined if a passengers gets denied entry.
4. There is an almost infinite number of combinations for countries of residence, departure, arrival; state of passports, visa types, etc. so it's extremely complicated. The airlines tend to use a service called "Timatic"
5. If Timatic says, you can't go, they will deny boarding. Unfortunately you can't easily check Timatic yourself: you need a paid membership.

The airline followed their stated process and you agreed to this process. The only potential angle would be to proof that Timatic actually returned incorrect information for your specific case. While not impossible, it's unlikely and it would take significant amount of time and money to proof any error on their part.

For the reasons mentioned in the other answers, you are not entitled to compensation. When they sold you the ticket, they could have assumed you held a Brasil residence permit, or a student visa, or any number of other situations that would have allowed you to stay 120 days without a problem.

For people who come to this answer, I am going to suggest a different solution. Do not modify your reservation to return earlier. Buy an expensive, refundable one-way return within the time limit the airline requires. In Brasil, once you have extended your tourist visa, apply for the refund. If, for some reason, your extension is denied, you can still recover some money by cashing that ticket in and advancing your original return date.