Whats the reason for the bullet making keyhole or oblong hole in target instead of nice round hole.
Short answer: the bullet is tumbling
Longer answer: if the bullet isn't spinning at a high enough rpm to stabilize it, it will begin to lose it's axis and will eventually tumble. There can be many things that cause this.
If we know more about the firearm being used, we can make a better analysis.