5.5 hr layover in San Francisco International Airport enough time to visit city?

It's do-able, but you won't get a lot of time in the city.

Unless you feel like paying for a taxi, your best option to get to the city is BART which leaves directly from the Airport. Presuming you're there on a weekday, BART leaves every 15 minutes (20-25 minutes evenings and weekends), and takes about 30 mins to the city. Allowing time to get from your plane to BART, buy a ticket, etc, you're going to be talking pretty close to 1 hour.

On return you're going to need to go via security which at SFO can take anywhere between 5 minutes and close to an hour, depending on your status/airline/terminal/time of day. Being an international flight, you'll probably start boarding around 45 minutes before departure time. So allowing for travel time, security, boarding and contingency, you're going to want to be leaving the city at least 2 hours before your flight time.

So out of your 5.5 hours, we've used up 3+ hours, giving you a little over 2 hours on the ground. Whilst that's certainly enough to see something, it's not much. It should be enough time for a quick wander around Union Square (BART to Powell st), Chinatown (BART to Montgomery) or The Ferry Wharf/Embarcadero (BART to Embarcadero) - but it's not enough time to safely get to Fisherman's Wharf or most of the rest of the city - at least not without catching a taxi.

BART will cost $8.25 each way. There's no return tickets as such - when you buy your ticket just make sure you put enough on it for the return journey to avoid having to buy a ticket when coming back.

SFO, especially the international terminal, is one of the nicer airports. The food is mostly from San Francisco establishments and the quality is good. I especially like Ebisu (sushi) and Burger Joint in that terminal. There are interesting art exhibits both before and after security. I've not been in a million airports but I've been in something like twenty in the U.S., and if I had to pick one to stick around for 5.5 hours, SFO would be it. Consider having a drink instead of rushing into the city for a couple hours. And I say this as someone who once took the light rail from the Saint Louis airport just to stick my head out and see the arch.

From my own experience it is often more entertaining to have a trip somewhere downtown, eat something nice or even meet some friend for 2-3 hours and going back than being stuck in an airport for that long.

Unless you know a good place to stay at the airport where you can sleep, have internet access or other means of passing the time, it is probably more entertaining to have a small road trip in the middle than to just be bored for 6 hours.

I normally have my bags checked, put my hand luggage in a locker, go to the city, have a nice dinner and then head back. Much more of an experience than burning the pattern of another airport carpet into my retina or spending even more money in the shopping area than a nice dinner would cost.