Don't feel bad- it's one thing if it's close to zero value, or at least, consistent in value and speed.
You didn't mention what bullets you were using. At 400 yards, heavier, high-BC bullets are going to start to make a real difference. I wouldn't use anything lighter than 168's if you're shooting Sierra's, the 175 is better if your rifle likes it.
The .308 is more than capable, 400 yards is but a chip shot for that round- my youngest son shoots his .308 Savage at 585 yards routinely- factory barrel and all on a 10" gong- and rarely misses when conditions are decent.
Here's where we shoot- and the tall berms- and the deep, long ditch from 585 to 1000 yards makes it really "interesting" when the wind is blowing. Usually we bag it if it's going to be over 15 mph.
http://manateegunclub.com/index_files/SateliteView.htm