Yes to what everybody said!?
I have the Pietta Remmie 36 and I use .375 balls in it. I haven't tried .380 because the .375 work fine in mine but - that is my pistol. Someone mentioned weighing your cast balls and if for nothing else than curiosity, I'd suggest you do that. Casting balls isn't hard as you know - but - there are variables. You have to get you mold heated up and as anyone knows when they are running balls, it takes a few until you get your rythm down and can be consistent in putting them out. The next time you cast your balls, try setting some aside from the first part of your run, some from the middle, and some from the end and then weigh them and see if the ones from the three batches are consistent with each other. True, a .375 ball isn't large but you can still get gas porosity in them - i.e. a small bubble or void - which will throw their weight off and can throw the accuracy off. I'm such a bad shot it doesn't make any difference.
But where I really start to notice it is in a larger ball such as the one I use in my flint Fusil de Chase. A swaged ball should be consistent in weight and if your cast balls are, then you can start looking at the other variables. Everyone has their own methods and what works for them. Another question I'd have is how are you measuring your charges (I'm assuming you are using BP - that's all I have experience with - I really do need to expand my horizons sometime and try a substitute). Are you using a standard powder measure or are you using a flask. Regardless of the pros and cons of using a flask, I've found over the years that when using a flask, for either a long gun or a pistol, I sometimes will throw an inconsistent charge if from nothing more than the difference in pressure of my finger on the tip of the spout as well as how I tip the flask. A lot of these guys have more experience than I and they've given some excellent things to think about. I look forward to hearing about what you come up with and if your accuracy improves. My Pietta .36 Remmie is a nice shooting pistol and I feel that the pistol is accurate - the main variable is my ability to aim it!
Good luck!