Is there a formula similar or close to Interrail for a family with (small) children?

I'm not sure exactly what your question is, because the InterRail pass you mention is clear about travelling with children and the costs:

http://www.interrailnet.com/interrail-passes/find-the-best-pass/child-discounts

There is no 'family ticket' for Europe-wide train travel beyond purchasing the suitable rail-passes (including the correct number of child passes for all children over 4 years old, as the link above explains).

If you are considering travel in one particular European country then you will probably be able to receive more specific advice about your options (as @Gagravarr explains about the great UK family railcard - there are likely to be similar schemes elsewhere in Europe).

I'm not aware of a European wide scheme, but if there is such a thing then hopefully someone will let you know! What I do know about is the Family and Friends Railcard in the UK.

With a Family Railcard, as long as your group contains at least one child under the age of 16, then you get 33% off adult tickets, 60% off child tickets, and children under 5 travel free, for up to 4 adults, up to 4 under 5-15s, and 2 under 5s per adult. (You need to buy at least one child ticket per adult, so if all your kids are under 5 you still need to buy child tickets).

These discounts even apply to advanced purchase tickets! You can buy the railcard online if you have a UK address, or buy it at the station before you travel. It's currently £28 for a one year railcard, or £65 for three years. In most cases, if you try to buy an expensive mix of adult and child tickets from a ticket office, they'll suggest the railcard to you. (Ticket machines won't, so you'll need to know in advance about it if you don't speak to a real person!).

Armed with your railcard, and booking some advanced purchase tickets (generally these go on sale 90 days in advance, sometimes with a smaller gap for weekend tickets if there might be engineering works), you ought to be able to travel all over the UK for not too much. If need be, you can book walk-on off-peak fares, and the railcard will stop them being too eyewatering... The rail network in the UK is pretty comprehensive, so you ought to be able to visit most places you'd want to, both cities and some parts of the countryside (national parks etc).

Sadly the Family Railcard doesn't get you a discount on the rail+sail tickets to Ireland, but those tickets are pretty cheap anyway, so that hopefully wouldn't be a big deal if you fancied a day or two in Dublin. The Family Railcard doesn't get you a discount on the Eurostar either, so you'll need to pay regular fares if you decide to come over that way. You might well find it cheaper to do the Rail+Sail from the Netherlands instead - the "Any Dutch Station to London" tickets are quite reasonably priced, and you can get a family cabin for overnight and then save on hotels.