Can I keep my old passport when I have it renewed and a new one issued?

Anecdotally, in my experience the "default" handling of old passports is for the issuing agency to destroy the old one. However, there may be a tick box on the passport renewal form (for example the Canadian passport renewal form) that asks:

Would you like the previous passport returned to you? If not, it will be securely destroyed by Passport Canada Program to protect your personal information.

Additionally, with regard to still-valid visas, there is a note:

Your previous passport will be cancelled at the time you submit your application and can no longer be used for travel. If you submit a passport for renewal that contains a valid visa, you must indicate on the application form that you want the passport returned to you and advise us that the visa needs to remain intact (inform the agent verbally if you apply in person, or include a letter with an application submitted by mail or at a receiving agent).

When they return an old passport, it is cancelled by clipping a corner or punching holes in it (not sure exactly what they do to preserve a valid visa, as I have not needed that).

I have retained all my previous passports, and I will continue to do so as much as possible. I know of people who have retired and needed to unequivocally prove the dates (decades ago) that they left a certain country or entered another one in order to be eligible for various kinds of retirement benefits. You never know when that information might become useful.

In the UK it used to be the case that you could ask to have your old passport back when applying for a new one. This is now what happens by default (unless the old passport is severely damaged). Cancelled passports are returned with the top-right corner cut off.

I still have all the passports ever issued to me.

This is arguably useful if you have visas for some countries that have not yet been transferred to your new passport. I once flew to the US with a US visa in my old passport.

It is useful in some cases when applying for a visa for travel to certain countries after a name change (e.g. due to marriage) as it can be used as supporting evidence to tie together several documents under different names.

In the US, your old passport is returned (separately from the new one) when renewing, and the official advice is to keep your old passport, as it is legal proof of citizenship (and perhaps has other uses) even though it's not valid for travel. However, as others have noted, this seems to vary highly from country to country.

In Poland it's customary of clerks to ask whether you want to take it as a keepsake, they only cut a corner off it so that it's clear it's invalidated. It's worth nothing in legal sense and you can safely throw it to the bin once you get your hands on a new one, although if you couldn't show your old passport before obtaining a new one, I think you'd have to formally report having lost it before you could apply for a new one.

In some countries there's no expectation that you would hand in your old passport when getting a new one. In addition, old passports can be crucial documents for proving citizenship. Australians born after 20 August 1986 must provide evidence that at least one of their parents was an Australia citizen or permanent resident at the time of their birth, both of which can be easily proved if the parent still has their old passport.

It all depends on the country you're a citizen of. Here in the Netherlands, by default your old passport is taken back when you pick up the new one and destroyed (shredded that is, the resulting trash being burnt in an incinerator).
You can ask to get the old one back, in which case it's clearly marked as invalid, big holes being punched through it (4-5 half inch holes in my experience).
Other countries as mentioned have slightly different but similar practices.

The idea to supplying old passports may be to be able to show a paper trail of past travel in case of trouble with your visa application (maybe one of them shows a prior US visa, making things easier, for example). For me, I'd not be able to show more than one old passport, kept that one because it contains a rather rare visa (country gets only a few hundred visitors a year from here, and most of them on a group visa so they have no visa stamped inside their passports).
Similarly my father kept a few old passports containing US visa to expedite applications for replacements (he had an unlimited, non-expiring, business visa, had to send an old one with the application as proof).
For most people, they'll have no old passports at all.

It's very common to keep your old passport.

I have all my old ones, both South African and New Zealand. They've clipped the corners and made it clear that they've expired, but I still have them.

The most valuable reason I can see for this was my UK visa. It was in one of my passports and valid until this current year. However, I got a new passport in 2010. At that point you have two options - either pay money to get the visa transferred, or keep and travel with both.

I opted for the latter, and from them on when entering the UK, I'd present my current one (to show I had a current NZ passport) and the old one to show I had the visa. The border staff had no problem with this - it was a common occurrence among expats that I knew.

If your country does destroy/take your passport from you, I would imagine in the scenario above you'd then have to go for the first option - pay more to get it transferred to the second.

I also have friends who have lost their passports. Therefore there's no way for them to present all their old passports, and if requested to do so on a form, would have to write that they were stolen/lost. It happens.

In Germany you can keep it but the clerks will use hole puncher to invalidate it. But maybe you "lose" your passport, get a new one and then suddenly retrieve the old one :)