Can dual US-Canadian citizens travel to the US with an expired US passport but valid Canadian passport?

Strictly speaking they need US passports to enter the US, which includes transit. However, they will be able to board the plane with their Canadian passports since Canadian citizens require neither a visa nor ESTA. Furthermore, US border officers cannot deny a US citizen entry to the US.

They can, however, delay entry, and if you encounter someone who is minded to make your life problematic for failing to use a valid US passport, they could delay you long enough to miss your flight. It would be safer to renew the US passports before the flight.

For the return flight, if there is one, you will probably need US passports. We had a report here of a US dual citizen who received ESTA authorization on her ESTA-eligible non-US passport even after disclosing her US citizenship on the ESTA application, but she seems to have been stopped at US preclearance in Canada, after having been issued a boarding pass. She had to get a US passport and return later to undertake her journey.

There are two parts to this question:

1. Will they be allowed to enter the US? Unlike, for example, European airports, North American airports don't have a transit area where you can move from gate to gate without going through immigration. So, your children will need to enter the US, even if just to leave again minutes later.

2. Will they be allowed to board the plane? The airline is responsible for checking whether a passenger will be allowed to enter the destination country. If the passenger is denied entry, the airline must transport them back at their own cost and in addition pay a hefty fine. Naturally, the airline wants to avoid both those things and thus will err on the side of caution. (Also, note that, should this happen, the airline will recover the costs from the passenger, i.e. your children.)

Re #1, it has been discussed multiple times on this site that US citizens must enter the US under their US nationality. Therefore, your children must enter the US as US citizens. Now, the good news is that being US citizens, they must be admitted into the country. However, the border control officer has every right (and duty) to make sure that they actually are US citizens … which they cannot prove. The officer cannot deny your children entry, but they can (and should!) do their due diligence and delay entry until they are sufficiently satisfied that your children are, in fact, US citizens.

Maybe the expired passports will be convincing enough, maybe not. Maybe, the US officer will ask a Canadian colleague to run the Canadian passports for some identifying information that they can use to look up your children in their system and convince themselves that they really are what they say they are. Maybe, they will require a long-form birth certificate.

Maybe, all of this will proceed fast enough for your children to catch their connecting flight, maybe not.

Re #2, the Canadian passport will probably be enough to convince the airline to let your children board. But, be aware that airlines use mostly automated systems to make this determination, and it could very well be the case that this automated system knows that your children are US citizens, that they are supposed to use their US passports and that those passports are expired. It is too dangerous for the airline to make any in-person determinations, or exceptions, or grant leniency. Also, the gate personnel are not trained immigration lawyers. They will rely on the automated systems only, and there will be no discussions with them, lest your children want to go through the experience of being dragged out of the airport by Israeli security personnel.

Unfortunately, this check is only made at the gate, since the airline cannot run the risk that something in the passenger's situation has changed between the time the tickets were bought and the time of boarding the plane. So, your children will only find out at the gate if they are allowed on the plane or not.

tl;dr: the safe route is to get new passports issued and use those to enter the US.

In a situation similar to yours, my wife (Finnish Passport) and I (US Passport) visited the US this past summer with our son (Finnish Passport and expired US Passport). We carried a photocopy of his expired US passport for easy reference if asked but used his non-expired Finnish passport for the airline and for the automated machine scan at border control in the US. We expected and were prepared to show and discuss his expired US Passport but we were not asked by the border agent.

I assume the computer and agent have access/knowledge of my son's previous entries and his US passport or that the information can easily be retrieved.

In all my experiences crossing the border by land from Canada or by air from other parts of the globe, every US Customs agent I have encountered has been intelligent, professional and focused on the job of protecting the US border (sometimes doing that with random or probing questions). Besides wanting to discuss it, I cannot picture that an individual agent would deny or restrict your entry or transit when your children have valid passports to travel with and their US citizenship is easily verified. Your children's' citizenship hasn't expired; just their documentation.

I would travel with their Canadian passports but take expired US passports (or copy) with you for reference if asked. Unless you have a reason not to, I would renew their US passports when you can since the future could be different.

(Note: I wanted to provide my experience on the issue but did not have enough reputation on 'Travel' to add this as a comment so added it as an answer instead)

It's not just immigration. It's the airline.

US-Canada flights are plentiful enough that you could simply get another flight if delayed by US Customs after physically landing in the US. Heck, a lot of US ports of entry are close enough you could just hop in a car and drive (e.g. JFK-Toronto). It's a bigger matter, however, if you are prevented from boarding in Israel.

The airline will not fly you to a destination unless they are convinced you are eligible to enter the country. They will refuse to allow you to board the plane.

There are a number of US infractions which are technically illegal, but with no penalty - like filing your taxes late when you don't owe any money. While Customs will let you in after dispensing twenty lashes with a wet noodle, the airline view will be "it is not legal", because if you somehow were refused, they have to pay the fine based on whether it is legal.

Hand over the Canadian passports, and mention that they are American citizens with expired U.S. passports.

I can provide anecdotal evidence from the mirror opposite of this situation: I am dual US/Canadian, but have never held a Canadian passport (I have always meant to get around to applying, but never have). I went to university in Canada close to the U.S. border, and travelled between them by air and land multiple times per year.

Like the U.S., Canada also has a de facto requirement for citizens to enter on Canadian passports, but also similar to the U.S., Canada will not deny entry to its citizens. Usually, I just end up showing Canadian border officials a picture of my Canadian citizen certificate or SIN card on my phone, and they admit me without hassle, stamping my U.S. passport.

I imagine it would work similarly in your situation unless you get a particularly unkind agent.

Just only travel with the valid Canadian passports. The US immigration officers will just think they are Canadian and grant them entry.