Sometimes I need to run errands after a workout (usually a hike) because, for example, the store is on the way home and I'm not going to change in my car or whatever. Other than that, though, I'm not fond of wearing workout gear unless I'm in the act of working out.
The What Not to Wear gang would agree with you. Shawn is also anti SIP (sweats in public - which was the workout gear of his college days). I also feel like a poser when I wear them, since I don't really work out.