I still wouldn't start at 320 grit. I highly recommend starting at 220. The fact is that 320 grit just won't remove enough wood to remove any scratches or nicks. And the question about how many coats of the Minwax Antique Oil is hard to answer. It depends on how big the pores of the wood are. Some Walnut is more open grained.
Don't be thinking that if putting a little Antique Oil on is good, then more is better. If you leave too much of the oil/varnish mixture on the wood it'll bleed out of the pores and you'll have shiny spots where you don't want them. So for the first couple of coats, apply the minwax oil and let it soak until the wood won't take any more (5 minutes max ought to do it, but before it gets real tacky and sticky) and then wipe and then buff the wood with soft t-shirt cloth. Then keep an eye on it so you can wipe off the tiny spots where the antique oil is bleeding out. There will be some of that. After maybe 2 applications of the stuff, which is one application per day, then you can start wet sanding. And try wiping sideways to the grain to try to not wipe all the sawdust/oil mixture out of the pores. The more coats, the more the shine, though it'll never be glossy. More than 4 or 5 coats may not improve things considerably, but there's no reason to not have a go at it. I did my Winchester 9422 and my Marlin 39A both like that and they turned out great. The Winchester had much better wood than the Marlin did, and it's the prettiest.