From the few I've talked with, and what I've read, most Special Ops types aren't all that enthused about handguns -- but carry them. (One SEAL I talked with said that for most projects, he'd rather carry an extra canteen than a handgun. It would, of course, depend on what task they're undertaking.)
The hardcore Special Ops/Special Forces folks seem to pick weapons depending on their assignment or mission, so having them stick with one set of weapons probably isn't realistic. They all have access to whatever they need.
You need to pick the "mission" and then pick the best weapons for that mission.