TL;DR

The rules have not changed (more about that below) and based upon your friend's narrative it was a fair cop. She was outside of the rules, no question about it, but this could have been handled with a better outcome and she could have been admitted. Based on her narrative they gave her about 4 chances to turn things around and apparently they all but bent over backwards (they were a whole lot more civilised than the other removals in our archives). She did not avail herself of those opportunities and instead dug herself in deeper. Now matters are worse.

Full Answer

Your friend was refused leave to enter under Part 9 of the Immigration Rules; all removals from port use Part 9 of the rules (or its restatement in Appendix V of the rules). The removal notice does not cite the specific paragraph/subparagraph but it will most likely be Paragraph 320 (1)...

(1) the fact that entry is being sought for a purpose not covered by these Rules;

The visitor rules

Your friend sought entry as a non-visa national who was making a short visit to the UK and so Appendix V of the Rules was engaged. These rules were drafted in the spring/summer of 2014 in a closely knit back-and-forth consultation between the Home Office and various members of the UK legal community. All told it took about 4 months and nobody on either side of the fence wants to do it again because it is too much agony. The rules were enacted in the Immigration Act 2014 and activated in April 2015. They have not changed since that time.

The landing interview

Specifically, your friend sought entry to speak at a conference (fee or honorarium or expenses-paid or whatever is all irrelevant). This is an activity that is not recreational and so "Visitors Appendix 3: Permitted Activities" comes in to play. What she proposed to do isn't found in the list of permitted activities and that's where she hit a rough spot...

Detention

She was placed in one of the mixed gender detention rooms and they gave her another interview (or more precisely, another opportunity to turn things around). That was her chance to get a temporary admission for a period of usually 2 - 7 days. This didn't happen.

Through what I count to be three separate interviews one of which was with a Chief IO. While a CIO must approve each removal, it's rare for them to engage the person directly. Something went terribly, terribly wrong and they were upset. It's easy to put all of this on the Border Force as demonic creatures given to sadism, but that proposition becomes awkward against the fact that they land 10+ million visitors a year without incident.

The removal

She was served an IS82, which is the form they use when someone does not qualify for leave-to-enter AND their behaviour is inappropriate. It gives them the option, but not the obligation, to have her placed on the Home Secretary's exclusion list (that's a bad thing). They removed her to NYC instead of her base on the USA's west coast. They didn't have to do that, there are flights out to the west coast all the time, but that was just icing on the cake. Minimal compliance.

Going forward

She has a removal which is computer linked to her biometrics, but worse is what's in her transcript. It cannot be expunged, but it can be relatively 'cleared' or mitigated by obtaining an entry clearance. It costs GBP 87 to apply for one, but I would emphatically recommend instructing a member of the UK Law Society with a practice area in prior removals who can word-smith an eloquent explanation.

Your questions...

If you hold a passport from one of the countries that normally entitles you to request "leave to enter" on arrival in the UK, and you're coming over to give a talk at a conference, what are the rules on if you need to get a visa in advance?

If what you propose to do appears in the 'Permitted Activities' section of Appendix V, you can get leave-to-enter at the border. If you qualify for the 'Permitted Paid Engagements' rule (also in the same place), you can get leave-to-enter at the border. You will need to engage with the IO in a landing interview to be sure you qualify, so bring your stuff.

And does it vary depending on if the conference organisers are paying for your flights/hotels or not? Or if they're giving you a small fee or not?

Not really. It's very common for such people to have their expenses paid. In theory you are not supposed to collect a fee; I don't think they pay a lot of attention to honorariums and the like (unless you do something to get them upset).

Or if the conference organisers are a UK or EU registered company?

No. The organisers need to be UK-based. Minimally that means there's somebody here who is operating under a 'management scheme' (in whatever definition you can provide that can be called a 'management scheme'). Again, personal impact and articulation skills. It's just that simple.

Note: it's my own estimate that roughly 1 out of every 2,600 non-visa nationals arriving from North America will be refused entry at a port. People have said here on this site that this figure is too remote to worry about. Read it here. I think comparison with lightning is a flawed argument. When a removal happens it's a big time disruption, so it's best practices to be prepared for your landing interview.

Note: the Chief inspects the detention facilities at LHR and etc to assure they are up to scratch. They are not comparable to the diplomatic lounge in IAD, but they are not the hell-hole of Calcutta either. The Chief’s latest inspection report as per their web-page was published 16th July 2015.

About the example, and why it doesn't show what it purports to

The refusal was based, at least in part, not on the speaking at a conference. If you read the refusal notice, you will note that the person in question was, in addition to giving a conference talk, running a workshop (IE carrying out teaching duties) for which:

Tickets have been sold for this workshop at a cost of £260 each for which you will receive 50% of the sales after expenses as well as a flat fee of £600.

As such, this refusal should not be seen as ruling on if (or not) it is OK to come to a conference in the UK and receive fees. Instead, it's more about whether you can be a paid teacher (even briefly) in the UK, while entering as a visitor (be that on a visitor visa or visa-free)

Answering the actual question

Beyond this, looking at the Visitor rules (which haven't changed, although the summary advice on gov.uk may have, although again note that the blog post never shows the result of clicking through on "permitted paid engagements") say that permitted activities (without being paid) are:

5 A visitor may:

• (a) attend meetings, conferences, seminars, interviews;
• (b) give a one-off or short series of talks and speeches provided these are not organised as commercial events and will not make a profit for the organiser;
• ...

and also:

1 The following are permitted paid engagements:

• (a) an academic who is highly qualified within his or her field of expertise may examine students and/or participate in or chair selection panels, if they have been invited by a UK Higher Education Institution or a UK based research or arts organisation as part of that institution or organisation’s quality assurance processes.
• (b) An expert may give lectures in their subject area, if they have been invited by a UK Higher Education Institution; or a UK based research or arts organisation provided this does not amount to filling a teaching position for the host organisation.
• ...

From this, we can see that giving a paid talk at a commercial event does not appear to fit within this space. In practice, I understand that something of a blind eye is turned towards speakers whose expenses remunerations turn out to be more than the expenses of the trip really were (although the problem with a "blind eye" is you never know if or when it will regain sight). But it seems the only occasion in which one could receive an explicit "fee" for speaking, and not be in breach of the visitor rules, would be if they were invited by a UK academic, research or arts institution.

There's some confusion about what is meant by

UK Higher Education Institution or a UK based research or arts organisation

understandably. I can't speak for exactly how this will be interpreted by UKVI, but as a UK native, my understanding would be that it includes:

• Universities and University Colleges
• Research Councils and the Arts Council, as well as any organisation dependent on them for funding and support.
• Professional bodies associated with Research or the arts - eg IOP, RAEng, The Royal Institution, etc
• Museums and non-profit performance groups

and that it would not include any clearly for-profit events companies, or simply private companies choosing to host a conference.