Clarification on travel with non-EU spouse to EU country of my nationality while living outside the EU

I've read https://europa.eu/youreurope/citizens/travel/entry-exit/non-eu-family/index_en.htm and some of the linked documents. I can't quite understand where does the following warning comes from:

If you live outside the EU and your non-EU family members accompany you or travel to the EU country of your nationality, EU cross-border rules do not necessarily apply and your non-EU family members might be charged visa fees.

I specifically cannot find any information about the above in this directive: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:02004L0038-20110616&from=EN#tocId44

Could someone clarify this for me:

what is the source of this warning - is there another directive?
and specifically - when they say "rules do not necessarily apply" - what is the meaning of "do not necessarily", where can I find the conditions and the scope of whether they apply or not?

visas eu family

Pictures about "Clarification on travel with non-EU spouse to EU country of my nationality while living outside the EU"

Clarification on travel with non-EU spouse to EU country of my nationality while living outside the EU - Three Flags on Flag Poles

Clarification on travel with non-EU spouse to EU country of my nationality while living outside the EU - Flags waving on old ornate city building on fine day

Can my non-EU spouse travel with me to Europe?

Under EU rules, you have the right to travel together with your core family members (non-EU spouse, children, dependent parents or dependent grandparents) to an EU country other than the one you are a national of. If you have moved to another EU country, they can also join you there.

Can EU family member travel to EU without visa?

After you receive your residence card as a family member of an EU citizen, you can freely travel to any other EU country. You can travel as a tourist for up to three months in a six-month period. You can also travel to many other non-EU/Schengen countries without a visa.

Do I need a Schengen visa if I am married to an EU citizen?

If you are travelling to the Schengen area as the spouse of an E.U. citizen, you should apply for the European Spouse Schengen Visa.

Can the wife of an EU citizen live in Europe?

If you want to join your EU spouse or registered partner in their country of origin and you haven't lived together in another EU country before, only national rules will apply to your situation. If you are in a same-sex marriage and you move to another EU country, your host country must recognise your residence rights.

Rights of EU Citizens (That Many Somehow Don't know!)

Sources: Stack Exchange - This article follows the attribution requirements of Stack Exchange and is licensed under CC BY-SA 3.0.

Images: Dušan Cvetanović, Dušan Cvetanović, Samuel Sweet, ArtHouse Studio