Theoretically, your exhaust can never be too big.
But you have to keep in mind that flowing air does NOT like change. So if you run a big pipe, you have to transition to that diameter slowly. If someone slaps a 5" downpipe on a stock turbo, you can totally expect to lose spoolup and power. The rapid expansion causes a restriction. The gases cool and slow down, making them tougher to move and this causes restriction.
Now if you upgrade that turbo and the outlet is larger, then a larger exhaust will benefit you more.
Ideally, the downpipe and the turbine outlet would be the exact same diameter for a bit, then transition slowly to larger pipe.
I would say to think of a Meg's exhaust tip for how you want your transition to look like-- a nice gradual cone.
Look closely at the Magnaflow systems. Magnaflow is one of the most highly respected names in exhaust engineering. Their system uses a smaller downpipe, THEN transitions to full 4". There is definitely a reason for this.
While all the guys spouting compressor theory are correct about the delta enthalpy, you have to remember that we don't have a turbine engine!
We have a PISTON engine with a turbo attached. As such, you can't disregard NA flow theory completely. The Pulsed nature of the flow in the exhaust (while less than an NA engine) is STILL there, albeit reduced by the turbine.
Also, keep in mind that a certain "IDEAL" exhaust is a compromise. Want the best spoolup? Be prepared to sacrifice peak EGT. Want EGT control? Understand your spoolup will be reduced. Sound familiar (turbine housings)?
Also, understand that as machines constantly in transition (fueling, timing, load, airflow) our engines determine that ANY and ALL exhaust systems are only "ideal" for a very limited range of conditions-- all else is a gross compromise.
Justin