Much of life in general is a gamble. The same is true regarding this issue. Odds are that you will never need your sidearm. Is it possible, yes; probable, no. Whether you decide to carry all of the time or not depends on how you like those odds.
There are many places I don't carry. I'm willing to accept the possibility and probability that I mentioned above. Others don't, and carry all of the time.
Are some of the folks that carry all of the time paranoid, nuts, or hoping to have an opportunity to use their sidearm? Sure, but I believe that they are in the minority. Heck, some folks here admit to carrying in their bathroom and/or shower. Most who carry simply want to be prepared for any eventuality, even if the odds of it occuring are miniscule.
I don't believe there is a right or wrong regarding when to carry. Just different degrees of comfort for various individuals.