Either cartridge would be adequate for cougars. Mountain lions really aren't all that large or tough so I'd imagine that any cartridge suitable for self-defense against two-legged predators would work equally well for cougars (the four-legged kind
). The 10mm will have a slight advantage with black bears due to its increased penetration when loaded with heavy bullets (more velocity and better cross-sectional density).
Another though to consider is ammunition availability, .45 ammo is simply much more common and less expensive than 10mm ammo is. Of course if you're handloading, that's not really an issue.
Neither cartridge will have prohibitive recoil. Personally, I find full-power 10mm (Double Tap 180grn XTP from a S&W 1076) to have less felt recoil than a .45 ACP (Federal 230grn HST +P from a S&W 1911 with 5" barrel). Be aware that you should probably re-think your ammo choices for a Glock as both of the loadings you mentioned use cast bullets, a big no-no for polygonal barrels. I'd suggest you look at some of the loadings from Double Tap with jacketed bullets, specifically their .45 ACP 230grn Controlled Expansion and 10mm 180grn Controlled Expansion as these would work nicely for double duty against both two and four-legged predators.