I played around with this awhile back. I did it for fun so I didn't document it really carefully--please forgive my "handwaving" in place of more appropriate mathematical rigor.
The first thing I did was find some loading data that compared light bullet loadings to heavy bullet loadings in the same caliber. Then I ran the numbers to find out if the additional barrel time could explain the observed difference in point of impact.
What I found out was that the difference in barrel time alone wasn't enough to explain the point of impact distances that are commonly observed. It turns out that you have to factor in some additional contributors before you can get numbers that compare favorably to what we observe on our targets.
Basically what it comes down to is that heavier bullet loadings in a particular caliber will typically generate higher momentums than the light bullet loadings for that caliber and therefore they also generate higher recoil velocity. That means that not only does the bullet stay in the barrel longer, it also means that the barrel rises more rapidly due to the higher recoil velocity AND it also means that the velocity of the muzzle as it rises in recoil is faster.
So the heavy bullet is in the barrel longer than the lighter bullet which means that the barrel would be pointed higher than it is for the lighter bullet even if it rises at the same rate.
AND the heavy bullet loading creates higher recoil velocity (more recoil) which means that even if the heavy bullet weren't in the barrel longer the barrel would rise more in a given amount of time due to the additional recoil velocity.
So now the barrel is pointing higher by the time the bullet exits for two reasons--additional recoil and additional barrel time. But that's still not the whole story.
Since it's rising faster due to the increased recoil the muzzle "slings" the bullet upward a little bit faster too. To understand this you have to realize that the muzzle is actually in the process of moving upwards due to recoil when the bullet exits. That means the bullet is moving upwards at the same velocity as the muzzle is when the bullet exits. Faster muzzle rise means more upward velocity on the bullet due to the motion of the muzzle at bullet exit.
So the additional recoil brings in two additional factors that move the bullet higher. A higher barrel angle by the time the bullet exits and a higher upward velocity of the muzzle at the time of bullet exits.
So it's not just the additional barrel time, it's also the additional recoil which contributes in two ways--higher muzzle rise (higher barrel angle at bullet exit) and the fact that the faster muzzle rise velocity is imparted to the bullet which gives it a little additional upward velocity besides the greater upward vector due to the higher barrel angle at exit.
Comparing a 135gr 10mm loading to a 200gr 10mm loading the difference in point of impact at 10 yards is about an inch if you only consider barrel time difference. When you factor in additional recoil and faster upward muzzle velocity you get a point of impact difference of about 2.5" at the same distance.