From a safety standpoint, I agree with the above. As long as you are not at the minimum you should be fine safety-wise. However, the only way to determine if your large primer recipe will give you max accuracy when used with small primers is to workup the load again just like if you were changing any other component.
If you are looking for a plinking load, sure, just swap components. If you are looking for max accuracy, the only way to get it is to work up the load again until you achieve max accuracy.