I would have no concerns shooting your cast loads. More surface area through the flash hole isn't going to increase the available pressure on the primer cup; which the cup is designed to handle. There are guys out there that use a ball bearing to swage the primer pocket smaller. I don't know if that would help you get your pocket small enough to eliminate the copper bushing. It may be easier to drill your flash hole from inside then you can use a tool that doesn't chew up the primer pocket edge (like your drill was) to remove any of the berdan anvil left.
Can you take a picture of just a primer fired in a case with no powder or bullet? It would be interesting to see if the primer backs out like on a regular case or if your sleeve backs out with the primer. That copper you are using is dead soft so there isn't any spring back. How many loads have you gotten before your pockets get loose?