This is what some other folks were observing as well, for example in Comparison of Single vs. Multiple GPU Instances
The workaround there was to run multiple instances to absorb these delays. I have not yet invested much time into figuring out what is going on there, but one day I will.