Dual national UK and Australia

No there is no problem whatsoever with the name change. You’re still an Australian citizen and if you have any evidence to document the name change that will be helpful. But you do not need a visa at all.

some countries require by law to enter with their country's specific passport if exists regardless of other passports in possession.

This Australian gov. dual national page states :

Entering and leaving Australia

All Australians, including dual nationals, should leave and enter Australia on their Australian passport. If you have a passport from another country you can use that for travel once you have left Australia. People trying to enter Australia as an Australian citizen but without an Australian passport will face difficulties and delays.

and

An Australian passport is the preferred and most conclusive proof of Australian citizenship when travelling. Dual nationals should use an Australian passport to enter and depart Australia.

As for your name - bring a proof of name change, but if you have registered as married in the Australian consulate ( as you should have in case of marital status change ) then this should not be a problem, and further more - you can, if you wish apply for a free new passport due to name change as explained in this page dedicated to passport related change of name

If you ...

marry or enter a registered relationship overseas and wish to assume your partner’s family name or add it to your own

you should present

an Australian RBDM name change certificate, because your marriage or entry into a relationship overseas cannot be registered by an Australian RBDM

( I actually admire how ordered and easy to find all that information is with a simple 2 sec. search - Kudos to the Australian related authorities .. )

And just as @ jcaron commented in other answer - your problem could be more with the airline than with the immigration officer in Australia and in fact the same above dual nationals page states ..

International airlines have an obligation to carry only appropriately documented passengers to Australia. Appropriate documentation for Australian nationals is an Australian passport. Appropriate documentation for a foreign national is a visa to enter Australia. If an Australian national attempts to board a flight to Australia without an Australian passport, airlines will likely be unable to verify their claim to Australian citizenship at the time of check-in and may refuse boarding

In that regard I must say that I also posses multiple passports with different names ( similar case to yours ) and whenever a doubt arise with any particular officer ( may it be airline or immigration ) i just show either the other passport or a form from the consulate stating legal name change .