Can I take a big jar of Nutella in my cabin luggage on a European flight?

No. Plain and simple. Nutella is basically a gel and honestly I tried once to ask 2 different airports by sending them an email, whether similar things were allowed. They said they are not. Anyway, Nutella doesn't cost a lot and you could try to "sacrifice" a glass of it, but I wouldn't do the same with others gastronomic specialities.

As stated in previous answers, Nutella will not be allowed in the cabin. However, you might find Nutella at the airport's duty-free shops (after you check-in and enter the boarding area). If bought there, then you can bring it in the cabin.

I took two jars of ovomaltine through Geneva airport last month. Very modern scanning machines. The items were noted by the inspector, but then passed through. I kept these in the fridge the night before which may have helped, but I doubt this would always happen.

Make sure the Nutella is really cold, so it's solid and not gel-like :-) Of course, transit in Amsterdam will be a pain...

More seriously, I expect the actual decision is based on what it looks like when going through the scanner, which is probably based on the proportion of water in the substance. There have been stories of some types of cakes, as well as cheeses, being rejected. The only way to know for sure is to try (of course, the results may vary from one airport to another, and from one operator to another).

Of course, if you want to keep the jar, nothing prevents you from dropping the succulent brown goo into the bin while keeping the jar. Have a spatula at hand to assist you with this.