I'd say that the EF 35mm f/2 IS USM
is significantly more hand-holdable. It's half a stop faster and it has optical image stabilization.
It's a beauty, and I do like my very similar 24/2.8 IS, but it's definitely much heavier (and much more expensive). It's also longer, which is what makes the 40 pancake so nice; I got quite a bit less of the 'glare' from random people that I might get if I'd taken something like the 35/2 IS, let alone if I'd taken the 24-105/4 in.
I do agree that there are plenty of better lenses, but man, making something so unobtrusive, cheap, light, fast(er than economy zooms) and with such great image quality is notable. You figure you'd give up something to get what Canon put into this lens, but I can't really find any major compromises; it really is the 'body cap' lens. It should be on every Canon shooter's camera when stored.