Almost like you, I've hunted deer with varying levels of success for 26 years with all legal equipment - rifle, pistol, bow and muzzleloader - at one time I would have used a spear if it was legal!
I tried scents and soaps and bags of leaves and dirt and anything else that gave me an edge. I hunted a lot and killed several deer. Then I read an article called Brut for Bucks and my deer hunting life changed.
This guy would put on Brut aftershave before going hunting! He claimed that the deer wouldn't know what the heck it was and, being curious, would come to investigate. At the time I smoked so I quit doing ANYTHING special to cover my scent, I smoked on the stand, I washed my clothes with everything else in the laundry and I even tried Brut! My success rate stayed about the same for a while. Then I read another article about a deer's sense of smell and how they could discriminate many, many different odors with each smell. The author claimed scents only gave them something in addition to the human scent to smell and didn't really cover anything.
After this, I still didn't refrain from anything and didn't use scents but I started paying more attention to the wind direction when I set up a stand. I would just go ahead and assume that any deer coming from downwind would smell me and I ignored that direction completely. I killed 24 deer, including 2 real nice 8 pointers, in three years using these tactics (limits are quite liberal in Tennessee).
The most important things for me, when deer hunting, are GOOD camo and keeping movement to an absolute minimum. I've had deer come to my tree and smell my flashlight or where I stood but they didn't seem alarmed unless they saw me move. I had one look up and stare at me for several seconds one time. I got amused and winked at the doe. I thought she would come out of her skin, man they can move fast when they want to!
Scents certainly don't hurt, as long as they aren't overdone, I'm just not convinced they help much and, for me, aren't worth the trouble.