I only turn mine on when I need it. My switch is a finger's reach away on the gearshift, so I just click it on and off as needed. No sense dragging the turbo down if I'm just gonna coast for a bit. And I don't idle enough around town to worry about incomplete combustion due to low temps, so I'd prefer not to waste fuel with it on at every stoplight.
I use it for warm up in the cooler months, but only to get it to 100-120* and then I start driving. Idleing with the brake on will push some of the exhaust back into the intake (valve overlap - when both the intake and the exhaust valves are open at the same time) and will foul the MAP and IAT sensors with soot. Ever since I pulled mine out and saw how bad they were I've kept my e-brake idleing to a minimum.
Downshifting with the brake on is harder on the clutch. There are springs in the clutch hub that will absorb shocks in the drive direction, but there's not much dampening in the coast direction. Plus you'll get more wear on the clutch when you let it back out. I'll click mine off or blip the throttle when I downshift.
I do use it pretty much every time I slow to a stop from 40mph or above. I'll downshift to fifth and click it on, and then click it off below 1100-1200rpm or so where you're not really getting any braking. I may downshift once more depending on load, but that's about it. And I really don't use it around town since the rpm's aren't usually high enough to do anything anyway except make noise.
That's good advice to cycle it whenever you drive it to keep it free.
Notice that all the guys that suggest leaving it on have autos?