Start with a 22 rimfire rifle, someone is always running a sale on one. Cheap to shoot, lets you develop and hone your shooting skills. Many times you can get a package deal rifle/scope together.
It's always nice to go to the range and shoot very small groups next to the guy that's shooting minute of pie plate at 100 yards. And of course the guy is usually shooting a loud-n-boomer-kick em hard magnum that he had no busness buying in the first place.
Something I did before hunting season was to practice with a air rifle, accurate beyond belief, can be shot at home if you have the space.