I did not like how my Taurus Tracker .17 HMR felt in my hands with the stock ribber grips that came on it. Replaced with Hougue Grips and it feels a lot more right. The ribber grip was just too small and uncomfortable.
One think I have to consider for my hands is that I shoot guns left handed, so for a semi-automatic, if it isn't ambidextrous it gets a lot less consideration.
If you have big hands, a small handgun might not feel right. If you have small hands, a big handgun might not feel right. I think that if you find a gun that you can pick up having never fired it and it just feels "right" to you, it will likely only feel better after a good bit of use. A gun that doesn't feel right, you may or may get used to.