Are there unseen dangers or is this prone to going awry?
In theory you might be able to make it work, but it introduces extra opportunities for errors to creep in to the process, and that can't be good. Perhaps of more concern is if your loading environment is so inclement that you don't want to spend too long there, is your concentration going to suffer?
Lots of people work with temporary or mobile setups that you can put away after use. You could do your charging and seating with a Lee hand press
or an arbor/toggle
press (usually only for rifle), neither of which require fixing to a bench, or you could make something mounted on a board that you can put away after a loading session. The only problem is that you may have to send the wife or girlfriend out to the garage so you can work without distraction.