That bullet weight (180) in your Mag will do as much damage to a deer as would your 308 with the same bullet weight. Just taking good aim is all that counts. Deer won't know the difference. When meat loss/damage is found. It's because your bullet has hit major heavy bone mass/structure i.e. shoulder or leg joints as in a poorly made rib shot. If you want absolutely no meat loss. Consider a head shot.
"Any rifle will shoot bullets through the same hole. You just have to be able to see well enough to aim right."