heck yes they will destroy your hunting. what deer wants to suck up a mouthful of ants with every bit of corn?
Two suggestions. First, move the stupid feeder 30 feet or more.
Second, make ant killing traps. Get a few quart sized plastic containers. Using a paper punch put dozens of holes in the top. run superglue completely around the top, gluing lid to base.
Mix a 3 to 1 quantity of plain borax, find it in any grocery store with laundry detergent, and plain cane sugar. add water until you get a heavy solution, then add about 1 ounce propylene glycol (pharmacy) per cup of to prevent it from evaporating. Antifreeze may work just as well.
Bury several containers completely up to the lid with no more than a cup of the poison in the container, and refill as necessary using funnel or turkey baster.
This will kill your ants quicker than a fly swatter.
This assumes that they will go for the sugar.