I would guess that the plastic plug is put in after the roll pin is inserted at the factory to prove that it was done.
Therefore I would hit the pin out in the direction of that plug so you wouldn't have to punch against virgin material. (so to speak)
That would create the least force and help save your seal.
Jay