In my reloads, I generally load lighter than max loads and neck size every reload and don't anneal necks. I clean primer pockets before reloading.
I found that when a few of the cases in a batch wouldn't hold a consistent neck size and there was a noticable difference while seating bullets, the accuracy got worse. That is when I dump the batch and replace them.
Neck size problems began to appear after I got at least 15 reloads out of every case with Remington and Winchester brass.
Recently, I switched to Lapua and Nosler brass as my standard and the necks seem to hold up better. Both have the case necks annealed at the factory. I am around 15 reloads per case and have not seen any neck size issues yet. Overall group size accuracy with the Lapua and Nosler brass has been slightly better possibly because the neck tension is better maintained from case to case.
I load for 5 rifle calibers and I have had only one case split (.22-250 neck) or any problems with primer pockets in more than 15,000 reloads over the last 2.5 years.