During Rifle season for Deer: Yes, as it is required by law(here).
During Spring Turkey Season: No, as it is not required.
During Bow Season for Deer: No(not required)
Basically, i won't wear it unless the law requires it. However, in my area there isn't much Turkey pressure, or pressure during bow season. If i hunted in a high pressure area, i might change my tune. I also hunt on private property, so that makes my situation a bit different.