You can spot orange a long ways. Since no critters I know of wear orange, hunters would assume it was another hunter.
Seriously, though. I wear it when I'm hunting denser areas, but not when I'm out in the small-game seasons or bird seasons. I also don't wear it during bow season.