Your thoughts on Colt are sound. Their 6920 and 6720 lightweight version are excellent choices for an M4gery. I'd also look at BCM's midlength ARs which are equal to Colt but offer the midlength gas system in a 16" barrel instead of Colt's carbine length gas.
The comment on PSA tends to be valid, they offer a lot for the money but I wouldn't go as far to say better than Colt. I have BCM, Daniel Defense and PSA and am impressed with what I got for a couple hundred less.