Well... we have a reasonably narrow window of opportunity to inject the fuel we plan to burn. It's the nature of the beast. We need to inject fuel as the piston is coming up to TDC on the compression stroke. How soon you start injecting that fuel as the piston comes up, and how late into the ignition stroke you continue to inject fuel affects power, longevity, emissions, noise, etc.
For the daily driver: Inject the fuel too early... and it's hard on the reciprocating assembly AND the injection pump. Inject it too late... and you're not doing much more than blowing smoke out the tailpipe.
BIG HP trucks are far from efficient. They inject lots of fuel very early and continue injecting it very late... the bottom line is that they're mostly interested in the massive quantities of fuel flowing through there. Back that up with the right amount of heat and oxygen... and you burn some of that fuel. There's a fine balance between a setup that's not too hard on parts... makes a boatload of power... and is still drivable on the street. Some folks on here have mastered that which is really encouraging to see.
The limit to how many RPM's a diesel will turn is dependant on how we inject the fuel (it's WAY different from a gasser). The mechanical limits of the parts that handle the injection process is the largest barrier. Getting an injection pump to turn 3000rpm (6000 engine RPM) isn't a big deal... but getting one with variable timing that will live at that RPM is a problem.
Common rail systems have a lot of hope yet. With seemingly limitless timing and the ability to control it very precisely... they'll live quite well at high RPMs. With a reliable high pressure pump able to flow more fuel at higher pressures... we'll be able to test the limits of the traditional injectors. Once we find out that we're at 90%+ of the duty cycle on the injector side... we'll have to figure out a way to get 'em to open/close faster... oh, and handle all that additional flow/pressure without buying the farm.
So, we're limited by the compression ignition boundaries set forth by our dieselheads of years gone by. ~6000rpm is doable now... how high do you need to go?
If we were dealing with gassers... we'd be injecting fuel even when the intake valve is closed... that gives you an example of the HUGE window of opportunity they have to get the fuel into the cylinder and burn it. We don't have that same luxury. (Those dieselheads that burn propane or methanol ARE taking advantage of the benefits of injecting fuel at a closed intake valve... it also comes at the risk of effectively advancing the timing a BOATLOAD - which is hard on parts. )
Personally, I enjoy the challenges diesels present and the differences that divide them from the crowd.
Matt