UK student visa legal situation for traveling in the EU more than 3 months without Schengen visa?

Your situation is not very clear (to me, anyway). As far as I can tell, your main problem is avoiding overstaying in the Schengen area.

If you are not a EU/EEA citizen and do not have a long-term visa or residence permit from the country you are staying in, you have to abide by the three-month rule, which is probably the choke point for your travel plans. The UK student visa basically does not make any difference here. Between August 10 (when you entered the Schengen area for the first time; is this correct?) and December (when you plan to leave to Mexico), you can only be in Schengen countries for a total of three months, which means you cannot just stay in the area without interruption.

As there are no systematic border checks, you might very well be able to remain in Portugal or travel to Germany (certainly overland) even if you overstay but you have to expect a lot of unpleasantness when flying home (potentially a hefty fine and a black mark on your record).

In practice, this probably means you will have to spend some time in the UK before December but going to other countries where you are entitled to stay could be a solution as well. For example staying a couple of months in Croatia or Romania – if you are indeed entitled to a visa or visa-free stay, which I don't know – could work as far as I know (those are in the EU too but not in Schengen yet). You could also go to the UK now and come back to Portugal after a few weeks, the three months need not be in one continuous chunk. If you can stay with friends and are flexible regarding the dates, this could be quite cheap (since you are trying to optimize your return fare, I am assuming you are on a tight budget). Leaving the EU entirely for a few weeks would work too, if you can find a cheap destination you would want to visit.

Alternatively, you could in principle apply for a long-stay visa if you are eligible for it but it does not sound like a very realistic solution. Many countries do not allow this at all if you already entered the Schengen area but some do, typically only for nationals of a small list of third countries. I don't know if there is any country where this would be possible for Mexican citizens. Even if there were, it would most likely involve extra paperwork and visa fees so it would not necessarily save money compared to flying back from the UK. There would also typically be additional requirements, e.g. proving you have enough money to support yourself.

Bottom line: If you can legally be in the Schengen area in December, there is no need to travel over the UK. If you cannot, the question is moot. The problem is avoiding overstaying, not being in or traveling through the UK per se. As you wrote yourself, you can only be in the Schengen area for a total of three months and, unfortunately, it seems that this is your answer right there, no easy way around that.