The handgun ammo with the best improvement for distance shooting out of a carbine is the revolver calibers, .357mag and .44Mag. But that generally means a lever gun.
All of the pistol rounds pick up some speed from the longer barrels of carbine, some benefit more than others. In auto pistol calibers, I only have one carbine, and thats a Tommygun (semi) in .45ACP.
Can't say exactly what kind of power increase there is (haven't chronographed it), but if you can manage the horrid spongey trigger, it will put 5 shots in one ragged hold at 25yds, and, using the ladder sight, ring the 200 meter rifle gong.
I imagine a 10mm carbine would do some nice things, but they aren't exactly common.
All else being equal (and it almost never is) bigger bullets tend to work better.