Do I have to carry my I-20 for travelling within the US?

There are generally no immigration checks for travel within the US. Your passport is an acceptable ID for TSA security screening purposes (which doesn't have anything to do with immigration).

However, to prepare for the very unlikely case you run into a CBP checkpoint or otherwise get checked by an immigration authority, you should carry your I-20 with you. Note that a US visa is only for entry and does not indicate anything about whether you are in status or not.

Not carrying your immigration papers could be considered an offense.

Per INA: ACT 264, Sec. 264. or 8 U.S.C. 1304, Every alien, eighteen years of age and over, shall at all times carry with him and have in his personal possession any certificate of alien registration or alien registration receipt card issued to him pursuant to subsection (d). Any alien who fails to comply with the provisions of this subsection shall be guilty of a misdemeanor and shall upon conviction for each offense be fined not to exceed $100 or be imprisoned not more than thirty days, or both.

Note that this seems to be an unlikely scenario. I couldn't find an instance where a legal immigrant was charged under this section. In particular, your Passport/Driver's License/State ID card is acceptable by TSA and they are not going to ask for your I-20. However, it might be a good idea to have it accessible on your phone in case you do run into an immigration official.

Federal regulations create the expectation that you keep the I-20 in a safe place rather than carrying it with you. See 8 CFR 214.4(f); paragraph (2) reads

An F-1 student is expected to safekeep the initial I-20 ID bearing the admission number and any subsequent copies which have been issued to him or her. Should the student lose his or her current I-20 ID, a replacement copy bearing the same information as the lost copy, including any endorsement for employment and notations, may be issued by the designated school official (DSO) as defined in 8 CFR 214.3(l)(1)(i).

Furthermore, the regulations concerning the carrying of immigration documents, 8 CFR 264.1, do not list the I-20 as one of those documents. It's therefore probably better to carry a photocopy of the I-20, along with a photocopy of the regulations to show to any ignorant officials who might incorrectly believe that you are required to keep the original I-20 with you.