What smokin54 said
And because the primers are seated just a little below flush, what is rattling is the base of the cartridges against the recoil shield. So, NOTHING is hitting the primers.
If somehow, you could jostle a gun holstered on your body hard enough that the inertia alone would set a primer off, your bones would be crushed long before.
In short, don't worry about it.