Don't get taken by surprize.
Depending on how good your peripheral vision, you have for the sake of practicality, up to about 180 degrees of vision. The 180 degrees you can not readily see or detect movement facing straight ahead is a major vulnerability. You must constantly be aware of who and what is going on around you.
There are specific techniques for breaking off a grab wearing a strongside belt holster and others, but they might be difficult to properly articulate in words alone. Better to seek a professional trainer for some one on one hands on.
An alternative or supplemental is to carry a knife accessible to the weak hand - either a small fixed blade or one-handed opening folder if legal in your state/locality. Trap with the strong hand, draw and use blade with the weak hand.
But bear in mind that if someone get's their mitt around the grip of your piece you'd better be able to turn it on 100% in an instant. Depending on their speed, agility and strength, your assailant might be able to pull you right off your feet, tear your whole belt and holster out of the trouser loops, or around your body etc - all by the grip of your handgun.
It can not be emphasized enough that if you spot trouble coming, even if you only get a few seconds, you have a far better chance of prevailing than if you are taken completely by surprize.