While I know you mentioned "at what point do you become a manufacturer?" if you give as a little more of an idea of what you would like to do, perhaps some of us could point you in a better direction.
No disrespect to Willie either, but there have been several people build AR's and similar in NJ, and its rare that I have heard of a problem, unless the firearm has features that are prohibited.
Since you have your location listed as South Jersey, you may be able to get some information from some that have/are building a firearm now. www.NJgunforums.com
seems to have several people building AR's and I noticed a thread in gunsmithing about a 1911 build too for personal use. They may be able to direct you to any legal issues that you may encounter. Just a thought.
As to you starting to set up a manufacturing I would contact the ATF personally, or through a lawyer, and discuss the issue with them, giving examples of what you want to do. You mentioned...
I am interested in machining parts to sell as parts. I just dont want to be in a predicament that I could be called a "arms manufacturer". I dont have the facilities to make barrels anyway. But there is a lot of room on lowers/frames, sights, grips etc.. As long as I can do so within the law.
There is some level of disagreement between some folks, but if you do work on a lower or frame, I would strongly suggest getting an 07 FFL. Building or doing the final machine work, would put you in the area of needing an 07 FFL. Bringing a finished lower/frame in, and doing such things as adding checkering, or doing other similar work may place you in the same area, since what you are doing is adding value to a product.
In my past discussions with the ATF, if a dealer takes a firearm, and adds something as simple as a new stock or install a scope, to make the firearm sell better, this can be understood to be a "value added product" and depending on the inspectors view may require a 07 FFL. Installing a stock or scope on to a firearm for a customer after it is sold, does not fall in the same area, and is considered gunsmithing.
As far as other small parts, I would still ask that you speak with the ATF and a lawyer with a detailed idea of what you are planning on doing, and see what they have to say.
Enjoy your day!