I personally have a 308 in the Tikka T3 Hunter brand. The recoil you get from a 308 does kick but it's fine for me. It's not to bad at all in my eyes and I am slightly similar to yourself as I shot 12 gauge when I was younger and I'd never buy a 12 bore its always 20 gauge I've owned because 12s kick like mules.
@ 100 yards the rifle was shooting 1 1/4 groups with sako shop bought 123 grain loads which was very frustrating. I reloaded a Nosler 125 grain ballistic tip boat tail load for the rifle and my god it pulled it right in to a 1/4 inch group. 3 shots at target, 2 were in the same hole and 3rd was in the 1/4 inch. That was when I relised it wasn't me it was the bullets and loads I had been using. All I am saying is that from my experience personally factory ammo may not give you the consistency that you are looking for. At least if you are reloading your own ammo then you can rule that one as the problem for the groups being slack.
The other thing is and this isn't directed at you but some people struggle to estimate the range of the shot without pacing it out so a range finder might be a handy tool to get the exact range your shooting and compare that to your ballistics's. As an example I was stalking with a guy last winter and his judgment on distance was terrible. I stalked a Red stag and he said " that stag is at lest 350 yards" and it was by no means that far. Max it was 200 yards out.