You didn't grow up or hunt in East Tennessee during the 70s did you?
I watched a guy empty his 94 thutty-thuitty at a deer, one time. One problem: He never pulled the trigger. Worse than that, he wouldn't believe it when I explained why the deer never fell--until he looked on the ground.
I had a friend who did the exact same thing (verbatim). When buck fever sets in, things like this just happen (more than you would think).
For the OP: Always approach downed game with your rifle (un-slung) such that it can be brought into play quickly if necessary. If the elk is down and can’t get up, but still alive, then I could see using a handgun for a coup de grace.
However, I do like handgun hunting with the appropriate handgun.