Removal of the panel is easy-- built-in hooks on the top that it hangs on with screws accross the bottom. Unscrew those remove door handle and I think there's bolt(s) in the armrest. Then lift and remove.
I have a rattle in mine only if the volume and bass are too high. Some day I'll look into it. Could be speaker, plastic vapor barrier, or lock linkages.