That bullet weight (180) in your Mag will do as much damage to a deer as would your 308 with the same bullet weight. Just taking good aim is all that counts. Deer won't know the difference. When meat loss/damage is found. It's because your bullet has hit major heavy bone mass/structure i.e. shoulder or leg joints as in a poorly made rib shot. If you want absolutely no meat loss. Consider a head shot.
"Do not follow where the path leads, rather go where there is no path and leave a trail."