A bullet traveling through the air never feels a side wind hitting it. If you are traveling at 1000 mph, what you experience is 1000 mph wind hitting you from the front. The natural wind of the air you are moving through adds, subtracts, or shifts the angle that this wind hits you by a fraction of a degree.
This "apparent wind" slows you down in the direction that the apparent wind hits you so if its direction shifts by a fraction of a degree, you slow down in that direction instead of the direction of your ground path.
Think about it this way and it becomes obvious why it's how much you slow down and not the time of flight that determines your course deviation.
As a bullet slows down, that wind angle also gets bigger, going from a small fraction of a degree to maybe a degree or so. The fact that slow bullets experience a larger headwind angle shift than fast bullets do explains why slow bullets drift more when the decelerations are equal.