Physical appearance and clothing, do they affect the passenger's chances of getting on the plane?

Your friend would not be denied on the basis you have given. The point of the search is to examine you and your belonging for dangerous articles, not to check if you have a Koran or if you have a poor sense of fashion. I will not speculate further on the reason for his denial to the secured part of the airport, but a messy beard and a book is not the reason.

In principle if you were wearing a provocative t-shirt, you could be selected for a more detailed examination (which would include removal of outer clothing and a hand search of your effects in private). If it was particularly provocative, the police may arrest you on the basis you are committing a breach of the peace.

You ask one main question and three sub-questions:

Can my physical appearance affect the airport security officers decisions?

Yes.

How is that possible?

Screeners are human and react emotionally. A provocative appearance is more likely to, well, provoke a response. This will be more true in places like the U.S. where airport security screeners are less well trained or disciplined.

I mean what would they put in their reports?

"Passenger looked suspicious."

that I was wearing a shirt saying "I <3 Bin Laden" ?

Yes, possibly.