For most big game, at rifle distances camo is not needed. Drab colors are just fine. I wouldn't bet on having much success wearing blaze orange around any bird, turkey or waterfowl. Most big game animals see color much differnet than human eyes. They see color, just not the same colors as humans. Birds however see the same colors as humans, only with far better vision.
I usually wear camo for everything though. I know it isn't necessary, but I already have the clothes and gear in camo, so why not use it.