Yeah, right now I'm looking for something small, but with a bit of oomph. The .204 will let me keep the target in sight, but still has some power. Once I bump it up to even a .223 or go with something like a 6BR, I loose that ability. That ability to keep the target in sight and watch the bullet strike is what really engages me. I've never seen that before in a centerfire. Then of course, I'm a complete noob.
You don't loose that ability with a .223 especailly if you are shooting light grained bullets. Plus you get that ability with larger caliber rifles if you add a muzzle brake. I personally don't care for muzzle brakes but if seeing the bullet impact the target was that important I'd add one to my rifle.
If you really want a rifle you can upgrade barrels and triggers on later I'd have to say a Remington M700 would be tops on the list. More aftermartet parts are made for it than any other action. More gunsmiths specialize in them as well. I liked the Varmint SF model but Remington has dropped that line but still has the VLS which would be my second choice.
CZ makes a very nice rifle and I love the single set triggers on some of their rifles. But I've only handled the CZ's and the biggest turn off to me on the varmint rifles is the magazine. When I'm walking with a varmitn rifle that is where I usually carry my rifle and I just don't think I'd enjoy the feel of that mag in my hand.
I'v only handled Rugers as well, and shot a couple of my buddies. I've never shot the varmint/target model. I can say that they build a rifle plenty accurate enough for hunting. I've only shot the laminated compact in 7-08 and the African in .416 Rigby and I know any .204 is more pleasant to shoot than that 416!