There is no way I'd leave a gun in a duffel bag at the gym, even with my eyeballs on it!! If you were female, you might realize just how fast someone can steal your purse...or duffel bag.
Why not carry a fanny pack to the gym? Or if it's not going to be on your body, lock it in your vehicle.