This happened to me as well back when I carried behind the hip. I ended up switching to the European mag style release (paddle under the trigger guard), found on Walthers and H&Ks. I also switched to appendix carry. Either would solve your issue.
Not saying you should switch your gun or carry method for just this reason, but its something to think about.