Transit visa via Muscat Oman airport and Yemen border go and back?

It depends how long your transit is for, all of you need visas if the transit is longer than 6 hours.

For Canadian citizens

TWOV (Transit Without Visa):
Holders of onward tickets for a max. transit time of 6 hours.
Visa Issuance:
Nationals of Canada with a normal, emergency or temporary passport can obtain a visa on arrival. They must have a return/onward ticket and a hotel reservation.

For Yemeni citizens

TWOV (Transit Without Visa):
Holders of onward tickets for a max. transit time of 6 hours.
Visa Issuance:
Passengers in transit for longer than 6 hours, can obtain a single transit visa on arrival for a maximum stay of 72 hours if they have confirmation from the airline that a visa has been approved before departure.

Both can get that visa on arrival, but the catch is they must be approved by Oman before you leave and that the airline submits this application for you.

Air transit visa:

To be granted by competent authority at air way passages to expatriate based on application submit by aviation carrier company entitle the holder to enter the country only once and resident for a maximum period of (72) hour provided that he should have enough money to cover his expenses and valid ticket to continue his travel.

Land transit visa and return

If you intend to travel over land to or from Yemen the air transit visa provided by the airline will not be enough. Although Oman has a separate Road Transit Visa you don't meet the conditions required for that (requires UAE visa, must exit through land) and you will require a tourist visit visa.

This question is very confusing, but I will try to answer it as I understand best.

You are flying into Oman, then want to cross by road into Yemen. Next, you want to do the return trip, start in Yemen, cross the border to Oman (by road) and then fly out from Muscat to Canada.

Here are the visa requirements:

For your trip to Yemen:

1. For entry into Oman, Canadian citizens can get a 30 day, single entry visa at the airport (source).

2. Yemini citizens will need to apply in advance at an embassy of Oman.

3. Canadian citizens will have to apply in advance for a visa to Yemen. (source).

4. The transit visa does not apply to you, because it is only for those that are traveling by air, within 72 hours - and its arranged by the airlines. Since you will not have such an itinerary, this does not apply.

For your trip from Yemen

1. Canadians do not need to apply for a visa, as one can be granted to you at the border with Oman (source).

2. Yemini citizens will need to apply in advance for a visa for Oman.