I take it you have large pockets. I appreciate that they're smaller than SLRs, but if you start taking accessories, different lenses etc, you're not gaining a huge amount in portability. I suppose if you don't ever want the size of an SLR, they might be the thing for you.
From my experience it was actually when I started adding accessories, lenses etc that I really noticed how much smaller it all is. My fairly small Crumpler over the shoulder bag now holds two bodies, a 7mm 3.5, 14mm 2.5, 20mm 1.7, 45mm 1.8, 100-300mm, 7 batteries (they eat batteries!), and a Gorillapod SLR. In short, all of my equipment and a staggering 14-600mm range. Back in my Nikon days I always had to make a choice when packing that bag (one less body for a start), my 70-200 2.8 never left the country and the missus had to squeeze the Gorillapod in her bag.
Once the 35-100mm is released next year I will be laughing (speed not confirmed yet, but apparently f2.0 or 2.8), as that's by far my favourite focal range and I would never have even contemplated taking my Nikon 70-200 2.8 VR on holiday with me.
By the by, my E-PL2 and pancake (14mm or 20mm) fits in my pocket just fine (coat in winter or shorts in summer), in fact it goes out every single day with me when we go for our evening walk.
