The minus of removing it is that many want it back in after tiring of the high pitched whistle. Problem is when reinstalling it the snap ring doesn't always fit well or folks don't reinstall it right. There have been several reports of folks finding the snap and/or silencer ring floating loose in their turbos with the potential for causing damage.
Remember if you are considering installing a free flowing air filter that the turbo whistle will be louder, you'll want the silencer ring then if you enjoy keeping your hearing intact.