The guy who runs the Tennessee Defensive Pistol Club is a rather large, robust gentleman who we call "Stuntman" - mainly because that's what he does when he's not raising that new baby his wife recently blessed him with. And, yes, we are aware that he is mustering a squad of "ringers" to come over here and give us "whut fer" but we're ready...(just kidding).
You have already guessed the secret to IDPA. Every "point down" from a perfect "a" zone hit will add half a second to your time. If you think the half second is tough, Bill Wilson actually proposed a change to a full second per point to emphasize his belief in the importance of center hits! Good thing he didn't get it.
Don't let the classifier taint your view of IDPA just yet. It doesn't really resemble a match scenario - it's just a standardized way to determine skill level. You'll find very few match scenarios that can be shot in 2 seconds (or even 10 seconds). My personal goal is to shoot as fast as I can and not exceed 1 point down per target in the string. Less is even better but "none" is too slow. Any target with a minus three is waaaay too fast.
In a scenario you'll have more cover to use, more reloads to do, more targets to shoot and more positions to shoot them from. You'll have to do some in tactical priority (nearest/most threatening first or slicing the pie from cover) and some in tactical sequence (one shot on each before any target gets a second). Some targets will have shirts over them to obscure scoring rings. There will be "no-shoots", hard cover and add-on props to indicate friend/foe (knives, guns etc.).
You'll find it slightly more challenging than the classifier.