Can I travel to the UK from the USA after visitor refusal from a few years ago?

You could certainly face delays, but given that it's a shorter visit, that your husband is a US permanent resident and that you have documnets proving ties to the US, you should be fine in the end.

Expect to be asked about your previous refusals, and explain how your circumstances have changed since to make you more low-risk.

As suggested in the other answer, flying via Ireland is an option, as you're entitled to enter Ireland as a spouse as per the EU freedom of movement and there are no border checks when arriving in the UK fom Ireland (except for occasional spot checks at ferry ports)

The 'golden' advice given out in case of any doubt regarding UK immigration is to apply for an entry clearance (aka a tourist visa). We've recently had a query about a situation similar to yours where the person in question had to waste several hours at the airport while having to explain his previous refusals. I cannot predict if it will happen to you, but applying for a visa will alleviate any problems you might possibly have.

A second stress-free option is to fly to the UK through Ireland to make things go as smooth as you possibly can. Since Ireland is an EU member they are obligated to let you enter the country as the spouse of an EU citizen, regardless of your former visa refusals (UK is also an EU country for now, but it doesn't apply EU spouse immigration laws to it's own citizens). And from Ireland you can fly to the UK facing minimal border controls - it's very likely that you won't even have to show your passport to the immigration officials.