Axle Oil Change - Remove Rust?

[Badunit]

Are you all sanding off the rust from around the differential before putting the cover back on? I noticed when I changed my rear axle fluid the other night that the rust layer was pretty thick and that it was a form fit around where the cover seal was. I sanded it down a little because the cover didn't look like it would necessarily go back on exactly the same as it came off. I just wonder if I should have sanded it more or if I wasted my time worrying about it in the first place.

 

[Midnight]

If it's not clean you can bet it's going to leak.