I have the same problem at times as well. There are sooo many variables that only you can figure it out through testing. It could be a heated barrel, it could be a clean barrel (I always run at least 3 foulers through before I start looking for accuracy). As you suspect, it could be scope issues (I had that, scope was loose and didn't catch it until I got home and was inspecting everything because I couldn't get a pattern). Could be trigger issues (my 14 has a heavy trigger compared to my other guns which can cause hand rotation on the pull). Could be a technique problem (your first shot, adrenaline overcomes improper finger placement or mechanics). I have gone out before and wasn't able to put the bullets where I wanted them to go, and the next time out, (without fiddling with the scope) put them tight and dead center.
I had a friend who was a counter sniper in the military and he taught me so much about the mechanics and technique of the actual trigger pull, that even though it felt right, it wasn't. Hand placement, finger placement on the trigger, breath control, etc... all factor in to where the bullet will end up. Shots 2-5 were good grouping, just off from shot 1, again, maybe shot 1 was the problem and not 2-5.