I, a Russian citizen, was denied a flight from Alicante through London without a transit visa

RyanAir is to blame, as you followed the rules precisely for UK landside transit with an EU residence permit. Unfortunately, passenger recourse against such problems is difficult. You could try to claim EC261 compensation for denied boarding, and/or seek passenger rights help from Spanish authorities at seguridadaerea.gob.es.

The rules are shown in Timatic, available at iatatravelcentre.com. I don't know why another answer incorrectly determined that you need a visa, although the rules are tricky and maybe the airline made the same mistake.

Here's (in bold) the relevant rule I get from Timatic with origin Spain, destination Russia, transit United Kingdom for less than 24 hours, citizenship Russia, residence permit from Spain. (I also get the same output with origin Russia, destination Spain):

Visa United Kingdom
Visa required.
...
TWOV (Transit Without Visa):
Nationals of Russian Fed. transiting through London: Gatwick (LGW), Heathrow (LHR) or Manchester (MAN) with a confirmed onward ticket for a flight to a third country on the same calendar day. They must stay in the international transit area of the airport and have documents required for the next destination.

Nationals of Russian Fed. making a landside transit with a confirmed onward ticket for a flight to a third country that departs before 23:59 the next day. They must:
~ have a common format residence permit issued by Spain, and
~ clear immigration, and
~ have documents required for the next destination.

RyanAir is a point-to-point airline. From the point of view of RyanAir you were flying to the UK, and thus needed to be in possession of the needed documents to enter the UK.

Rynair's T&C's mention the following:

Article 7 - Refusing to carry a passenger

7.1 We may refuse to carry you or your baggage on any flights operated by an airline of the Ryanair Group (Ryanair DAC, Ryanair UK, Buzz, Laudamotion and Malta Air) if one or more of the following circumstances apply, or we have good reason to believe that they may apply.

(...)

You may try to enter a country you are joining a connecting flight in or do not have valid travel documents for;

The other answers all address the matter of whether you are considered in transit or not when doing a land side transfer in Luton. This may be beside the point. You were refused based on Ryanair's own T&C's.

What you should do in the future is not to self connect at all, or at least not self connect in a country you are not allowed to enter. If you had had your Alicante - Moscow trip on one booking there would not have been this issue.

I want to know who is to blame and what to do in the future in such a situation.

As mentioned on If an airline erroneously refuses to check in a passenger on the grounds of incomplete paperwork (eg visa), is the passenger entitled to compensation? by JBentley, and relevant since Alicante is in the European Union, you're entitled to compulsory compensation:

For flights originating from or destined for the EU, the Flight Compensation Regulation (Regulation (EC) No 261/2004) provides for compulsory compensation from airlines which deny boarding. It cannot be contractually overridden.

Welcome to the club.