Thanks for the follow up.
Just to reiterate though, and I hope this makes sense, the real issue isn't necessarily the extra play, it is the fact that the hitch pin is no longer in a proper "sheer" scenario. If you think about it, the adapter isn't mounted to the hitch nor the stinger/ballmount, it is just floating there in between the two. From the pin's perspective, the adapter might as well not even be there. The forces applied to the pin from the stinger, are about 1/4 inch away, on each side, from where the pin is supported by the hitch.
With a 2.5" stinger in a 2.5" hitch, there is no appreciable distance between the stinger and hitch, placing the pin in a proper sheer scenario, but when using an adapter with a 2" stinger, that 1/4" gap (x2) allows more leverage to be placed on the pin so it can flex and bend. This is what causes the wallowing of the pin hole. You'd likely find your hitch pin is no longer straight and will be slightly bent.