Can I drive my British car in the Netherlands using my Dutch drivers license as a UK resident?

To accomodate the people living in/around Baarle-Nassau, there's bound to be some special rules between Belgium and the Netherlands, but I don't know them - but in general the rule is that a car should be registered in the country where it is used. The point is that taxes associated with the car are meant to contribute to the society that has expenses (e.g. maintainance of the roads) from it's use.

Of course there are rules for passing through and vacationing in other countries, but as soon your use doesn't match those, the rules get strict. An example: I live and work in Copenhagen, but have colleagues who live in Malmø, Sweden. They are allowed to drive their Swedish registered cars to/from work, but (as far as I have understood) they are not allowed to use it for trips wholely within Denmark, i.e. they can go shopping in Danish stores on their way home from work.

When you have to stay in a different country for a limited amount of time there's a possibility to get permission for using your car while there. When I was 9 (so too young to care about the tax details), we lived in France for ~3 months, because my father had to attend some course together with some of his colleagues, some of which brought and used their Danish registered cars (my family didn't have a car) - they probably paid some tax to France (and possibly got some tax return in Denmark).

Returning to your cases.

The Dutch authorities were probably right when they issued that fine, and you should look into if your planned stay in the Netherlands is so long that you need a permission to bring and use your car while there.

What matters is the country of residence. You did not write that explicitely but it sounds as if your friend might have been a Dutch resident (possibly "borrowing" his or her parents' car regularly). I have heard similar stories about German citizens driving a German car with a German driving license (not even close to the border), the issue is that the Netherlands has quite high taxes for cars so people are tempted to circumvent them by using a car registered abroad long-term.

So, in principle, what you need to have is some evidence that you reside in the UK (although even that might not be needed as it should be easy for the police to check whether you are registered as a resident in the Netherlands).