Are non-Muslims allowed to enter mosques in Malaysia?

Yes, non-Muslims are allowed to enter (at least some) mosques in Malaysia.

See user casvaart's answer for a local's perspective.

Having been in Malaysia, I can attest that visiting a mosque is possible for non-Muslims. There were no restrictions as far I could tell and I have entered at least a dozen mosques principally around Kuala Lumpur but also some in southern provinces.

In some cases, there were signs indicating not to take photos within the prayer hall itself but the outer part of the mosque can be photographed.

Generally yes, but you might be asked to wear appropriate clothing (e.g. no exposed skin). But even then you might be given a gown to cover yourself with during your visit. At least that was the case at Putra Mosque (seen here)

I can't speak for every mosque in the country but certainly from my experience of visiting a few mosques in KL and Malacca it's a case of "Sort of".

The mosques all had visitor areas where nobody had any issue with me, a white guy in jeans and a plain t-shirt coming for a look around.

Signs said no shorts and no shirts with pictures of skulls (or other wild things) on them but I encountered zero issue, they're used to tourists.

However...this was only to see a visitor area. To go into the main part of the mosque itself where the worship takes place; the big two in KL had signs explicitly saying this was not allowed for non-muslims.

If I remember right the national mosque had set hours when non-muslims were allowed to take tours but generally it wasn't allowed.