It does always come down to what works best for you.
But maybe when you kept the front sight on the target, during the mag change, the gun was leveled too much.
Bringing the gun straight back to the chest, angled about 45 degrees, seems agreeable, both for the mag change and getting the gun back on target fast, without having to be a contortionist.
Walt Kelly, alias Pogo, sez:
“Don't take life so serious, son, it ain't nohow permanent.”