Travel without visa in Canada

Apparently, you do need a transit visa and do not qualify for the transit without visa program (which is not a document but a regulation allowing some people to transit without any specific document beyond their passport) as it's not open to Malaysian citizens.

You should either change your flight or apply as soon as possible and hope the visa application is processed quickly. You also risk being denied boarding in the US and losing your ticket entirely as airline personnel should check your right to transit through Canada.

I don't know which “fine” you have in mind but fines are typically not intended to replace visas and you won't find any Canadian personnel to deal with this in Portland. US border agents cannot grant you the right to enter Canada so you will most likely be stuck there and never get to the point where you could pay a fine in Canada.