IMO, the deciding factor is range. under 400, use the .243 for accuracy, over 400, use the 06 for range and FPE.
A .243 is not inherently more accurate than an '06 within 400 yards, so that argument cannot be made.
Is a .243 enough for deer? Sure it is, and it is a dang good deer cartridge. However, if you go for any game larger than whitetail, you are pushing the limits of the cartridge. I personally would not take one elk hunting, there are better options available for that purpose (with the '06 being one).
My Winchester Model 70 will shoot 5 shot groups the size of an M&M at 100 yards, but my Remington 700 .308 will shoot groups the same size at 200 yards. However, if I am going after deer or antelope, I still take the .243.
Why? Because even though the .308 is *slightly* more accurate at extended ranges, .5" will not make a difference in a kill shot on game of that size out to 300 yards; and game up to 200 lbs is really where the .243 is the most useful. It makes clean kills, and doesn't destroy too much meat.