It's always hard to second guess these things over the internet.
Anyway, if the gun is ejecting fine when firing the gun keep in mind the fired brass is much shorter then a loaded round.
If the gun fails to eject loaded rounds the ejector may be to long or the angle on the ejector nose may be wrong.
Keep in mind Colt's fix for this that you see on series 80 Colts.
Colt opened up the ejection window by lowering the height of the port and also cut a scallop at the front of the port to clear the bullet of a loaded round.
This may be whats needed if the gun fails to eject loaded rounds.