I have a buddy that owns a Batterys Plus store here in Ak. He had to go to a month long training class about batterys and such and he relayed a very interesting story to me that relates to this very well.
What they did was take a dual battery set up (like our trucks have) and hook up one huge semi battery and one little punny battery for some rice rocket car. (both batts are 12 v). They used this to run some 12v contraption that put a good load on the batts. They hooked it all up and left it over night. In the AM the contraption had stopped working and the batts appeard dead. They unhooked the little battery and hooked a battery load tester to it and it was dead. Then they hooked the tester to the big battery and it was fine. They hooked the load contraption to the big battery and it worked fine. Moral of the story, a two battery system is only as good as the weakest battery.
So I would say just run the yellow top by its self or leave the batts alone.