Maybe it has nothing to do with the brake and you've got a scope that's a little flaky.
Scope adjustments typically are direct drive in one direction but rely on spring tension to return the adjustment in the opposite direction.
So when you dial it in one direction it takes a few shots for the reticle to settle because that's the direction in which the spring tension is driving the reticle and maybe the reticle is a little sticky. It might take the bump from a couple of shots to get it to move all the way. When you dial it in the other direction the turret is driving the reticle directly so the reticle moves exactly as it should.
Try tapping the scope after you make the adjustment and see if you can get the reticle to settle before shooting.