Is it illegal to take a piece of the Berlin Wall?

There is a hotel in Berlin that lets you chip away your own pieces of the wall and take them with you as part of a package deal. So there do still exist legal options for you to take your own piece home.

This contradicts travel etiquette more generally. (Barring some more specific local ethic saying it's okay, but that doesn't seem to exist here). The ethic is known as Leave No Trace (wikipedia) and is developed for respecting the outdoors, but in my opinion, perfectly well applies here.

Taking a pebble or stick or leaf when on "sacred" grounds, whether it be a historical monument or a U.S. national park, is in violation of this ethic. Historical monuments should be preserved for hundreds or thousands of years and enjoyed by thousands or millions of people. "Taking a pebble" is simply unsustainable at that scale.

The issue is not whether there are enough pebbles to cover all the ravenous tourists. The issue is that tourists are engaging in the "take" ethic in the first place, and in seeking something for yourself special that you alone can enjoy, you are, naturally, seeking something that others cannot take. The pebble is an interesting example because it apparently is unique and hard to obtain, but subtle enough that maybe a tourist can get away with taking one. If pebbles were okay to take, you would know that already. It would be in the tour books or notated at the exhibit, and, possibly, be no more special to you than a photograph.

So, kudos to you for researching this cautiously. I just want to emphasize that the most general answer is don't do it, barring specific permissions from the local authority responsible for preserving the site.

The gift shop at the Checkpoint Charlie / Berlin Wall museum sells small chunks of the wall in various forms (e.g. encased in plastic at the end of a bookmark) for reasonable prices, like a couple Euros (at least in 2005 when I was there). I'm possibly mis-remembering which gift-shop I got the bookmark from; I gave it as a gift so I don't still have it.

The museum is worth a visit; there's some neat stuff ranging from homebrew radios to Stalin propaganda. Also stuff about the various iterations of the wall, and news reports of the first person to attempt to escape, and other notable things.

Also very cool was a tour of a sentry tower, but this was a private individual who (IIRC) was the brother of the first person to be killed trying to escape East Berlin after the wall went up. He didn't speak much English, but apparently spends some of his time passing on the history of the place to tourists. I got the impression that this was just something he does personally, not getting paid for it by a museum or anything. This was 10 years ago, and he wasn't a young guy then.

Back to the souvenirs: At the higher end of the price range, there were paper weights (again clear plastic) with bigger chunks (larger than pebble size). The chunks all had a piece of the surface visible, usually with some graffiti, so you could tell it wasn't just any random pebble.

If you really wanted the bare chunk, you could just buy a bookmark and crack the plastic. Or maybe find a different tourist shop than the one I visited, if this isn't unique to that museum.