Seems worse for me right after oil changes most likely the breather hose, look under the hood for leaks but my bet is the breather hose.
I get used to turning off the blower fan when it's bad at traffic lights, just one of the joys of owning a diesel.