The majority of people are so wrapped up in their daily lives they don't pay attention. Even if a person did see a bulge on your side chances are they'll think it's a cell phone. Many popular cell phones are a bit larger in the first place and when you get a Otterbox cover or similar with a belt clip it becomes quite bulky.
I'm a pretty "weighty" fellow and I carry IWB at 4 o'clock. I've carried with different clothing and in numerous places where it would be easy to be spotted if it was going to happen. You don't have much to worry about honestly.