More to what Chris said, check the idle screw itself. I had the problem of my idle being low one day, and high the next. I figured out that the locknut on the idle set screw wasnt tight. My only explaination of what happens from there is that if you goose it, the vibrations are enough to move that idle screw around.
Whats even more, is it always seemed like if it ever was low, it was when I first started it up. My only explaination here, is that when you shut it down, theres once again that vibration that makes the screw move, but now the throttle lever is being pushed against it so the only place for it to go is backward.
I tightened down that locknut, and never had the problem again.