Saddle soap will dry it even more.
It is perfectly proper to oil leather. The problem comes with our modern concept that "more is better." Oil sparingly and then wipe even that off. If you talk to anybody who works with tack as a profession, you will get told how to do it properly. You are not giving the holster a bath.
I have used neat's foot oil and I have used the more modern oil waxes. Both work fine. When I was growing up I was exposed to some old timer drovers and cowboys. They carried neat's foot oil with them so they could clean their tack ehen necessary. Most of them carried a 2 ounce bottle and refilled it once a year.
While in high school I was around leathersmiths at Hamely's Saddle and Tack Works. They all told me the same thing. Touch your finger to the oil and spread it out with the heel of your hand. A drop should do a holster. Two drops will do a pair of boots. Bridles and reins need so little that it hardly counts. Do your saddle skirts and your hands will then be oily enough to do your bridle and reins.
Armed and Safe: Not just a theory
If it time to bury them, it is time to dig them up.
Remember, "Behind every blade of grass."