Do I need a Transit visa while travelling to India via Frankfurt on H1B extension

The general rules are explained in Do I need a visa to transit (or layover) in the Schengen area? but it seems you have already figured them out and the answer is still somewhat ambiguous for your situation.

The website of the German Missions to the United State is clear on one thing, the approved I-797 is not enough:

Please note that the advance parole (I-512) and the approval notice (I-797) are not valid documents for visa free airport transit on your way back to the United States of America.

The text also implies that someone holding an I-797 could fly from but not back to the US (otherwise it would say it's not valid, period), presumably on the basis of the “return from the USA after having used the visa” rule you mentioned yourself.

One issue is that some countries have or had rules like “within 6 months” of the entry into the US or not more than X days after the end of the US visa's validity. So the “return after having used the visa” clause might or might not apply and unfortunately I don't know precisely how it's handled in Germany. If the exemption does not apply, Indian citizens do need an airport transit visa, even for airside transit, as you probably know.

In practice, they are not always the most helpful but you could try to contact the consulate and see if they can provide details. The airline will also have to decide whether they let you board or not (if they think you don't have the right to transit, they could get in trouble and your trip would end before your plane even leaves the US) so it might be worthwhile to ask them what they require. Hopefully someone there knows the rules a bit better than we do.

I flew back from US to India via Frankfurt and no transit or any other visa was required, because transit passengers stay in the secure zone before immigration.