I wear contacts too, and take them out every night. Luckily, I'm able to see well enough without them to identify objects in relatively low light. I also have a flashlight mounted on my HD gun.
If you can't see two feet in front of you, you cannot reliably identify your target. I wouldn't be using a gun as my primary defense weapon if that were my situation.
Maybe a pair of those "sports" glasses with the elastic strap would work. Practice donning them like you would practice reloading in the dark and they may not be the liability you think.
Good luck!
-Dave