Edit: Since you have a carry permit, keep a loaded handgun with you.
Any situation that would allow you to retrieve the ammo and load the rifle is a fairly slow-developing situation, and one that you should in theory be able to escape - given that you are in a vehicle.
OTOH, if you break down somewhere, you could load the rifle while you wait for help, I suppose.
But if you were run off the road and your vehicle disabled, I can't see you having time or wherewithal to get ammo and load up. If your handgun can't buy you time to retrieve and load the rifle, you weren't going to get the rifle into action anyway.