Overstayed of 5 months and going back to my country with one way ticket

Unless you are a recognized family member as defined by the Directive 2004/38/EC of the European Parliament and of the Council of 29 April 2004 on the right of citizens of the Union and their family members to move and reside freely within the territory of the Member States, you are subject to the rules of a Schengen visa.

Family member: this covers, for instance, the spouse, a partner in a registered partnership with an EU citizen and direct descendants under the age of 21.

Repurposing the answer from What is the penalty for exceeding the 90 day limit in Rome on an US passport:

The immigration officer has great deal of discretion concerning penalties for the overstay, and may:

• let you go with a stern warning (but given the circumstances, and the alarm over illegal immigration to Italy, unlikely)

• impose a fine of several hundred euros

• flag your passport, making it difficult to obtain visas in the future

• ban your altogether for a period of years— not just from Italy, but maybe from all the countries in the entire Schengen area

If they do nothing, when can I come back again on a tourist visa (90 days). Will I need an invitation letter?

Mauritians don't need visas so you can return, not guaranteed though. Take note of that. You will definitely need an invitation and other proof of strong ties to your home country.

What if, when I go back to Mauritius, I marry my boyfriend there. Will I be able to enter like an EU citizen spouse or am I still forbidden entry?

Yes you will be allowed