Yes, you make a good point. In theory what you wrote could be accurate but the difference would be miniscule. Most defensive drivers accustomed to driving with an exhaust brake watch ahead for traffic changes and allow their vehicle to slow slightly with traffic w/o defueling and activating the EB.
If you are talking about someone who is seriously driving for economy by relying on coasting as much as possible like the old Mobil Oil fuel economy competitions of many years ago I'd agree with you.
I think any possible savings achieved by not using the eb would be spent instead on slightly accelerated brake pad wear.
I've driven three Dodge-Cummins Rams about 600k miles with exhaust brakes. I'm a believer!