The wedge on a Colt cone is there to hold the barrel on. The barrel should fit whether the wedge is hammered in or thumb pushed in.
It is not to adjust the cylinder gap.
If the cylinder gap changes with pressure on the wedge then the arbor is the wrong length. Read Pettifogger's articles on proper tuning of Colt style cap and ball guns.