What is the size limit of the accessory bag on KLM flights?

As of 2017, their website has actual figures on it:

1 accessory, e.g. a handbag, briefcase or laptop, max. 40 x 30 x 15 cm (16 x 12 x 6 inch)

This is 18L, so a 35L rucksack unless very empty is unlikely to fit.

The official KLM baggage page which you found says:

1 accessory, e.g. a handbag or a laptop

Which isn't that handy, but there's an official answer on the KLM Facebook page that expands this:

The accessory can be a handbag, briefcase, camera, small laptop, or another item of similar or smaller size.

There is no definitive rule for actual size that I can find (even from partner airlines), but basically a suitcase is a good measure. If your 'accessory' could reasonably be molded into a suitcase sized space you'll be OK.

As mentioned in the comments on busy flights they'll ask you to put the smaller bag under the seat in front of you. So it'll have to fit under there, and leave you enough room for your feet! In my experience a suitcase or similar size laptop bag fits under nicely in an upright position and leaves a fair amount of space (although some KLM planes have restricted space under the seats as the unit driving the entertainment system is there for some seats).

Trying to take two obviously large, full, bags will likely not be allowed. They often do a quick check at check-in (not so much at Amsterdam which is automated but at other airports) or you'll be stopped at the gate and asked to check one. If you're stopped at check-in you'll have to pay, at the gate it's up to the staff there.

From experience on similar routes (been a long time since I did that one), they're generally quite relaxed about the interpretation of accessory as long as you're not being ridiculous. Large bags can go in if they're empty and go be packed down. However it does depend how busy the flight is, and it makes life easier for everyone if you don't overdo it.