There is some rubber around the bearing itself, to isolate it from the frame. As long as the bearing is on the shaft tight, and the bracket is good and tight, I wouldnt worry about it. SO, some up and down movement if you, say, grab the driveshaft and go up and down with it, is normal.
Daniel