There is a possibilty that the back end of the chamber was cut slightly oval and that could be leading to the problem you are having.
It is quite rare for this to happen, I have only come across one rifle that had this problem, it comes from the barrel not running true (excessive run-out) in the chuck while the chamber is being cut.
Usually a little run-out will just cause the chamber to be slightly oversize and remain round as it should be.
As long as the headspace is correct you shouldn't have any safety issues with it the way it is, the rifle I saw with this problem that I mentioned earlier is still in use today with the barrel unaltered.
It will probably put a tiny bit more stress on the brass and slightly shorten the life of it.