Double Naught, my experience is also exactly the opposite of what you describe. I don't factor my time into the cost of reloading for many of the reasons previously described, which in the end means it costs me about half as much to reload my own ammunition. In my own case then, I shoot twice as much if not more. And as far as the designation between reloaders and shooters, about half the guys I shoot IPSC with all reload so they can afford to shoot more.
And I also golf, so I'm not sure your illustration about guys trying to save money by scrounging the rough for balls is entirely accurate. Searching for lost balls is fairly time consuming, and usually what you find is of questionable quality. Reloading doesn't take me a ton of time, and I end up ultimately with ammunition of higher quality than what I can buy. So maybe a better golfing illustration is guys who start making custom clubs because they think they can build something better than what you get off the shelf. And here again, I've purchased custom clubs for myself which ultimately cost less than major brand names and are more ideally suited to my own swing.
The point in all of this is to say that some of us believe we can often get a better product at a cheaper price by doing the work ourselves. And on top of the superior product, we enjoy the process. The combination of these reasons is why we ultimately reload.