Wherever the sights are pointing, you should at least have a group somewhere, otherwise it's the shooter and not the sights.
This gets you to 'accuracy' and 'consistency'.
Usually, in my experience, firearms are consistent (i.e. if they aren't moved, they'll put repeated holes in just about the same spot), with some exceptions I've run into (primarily loose sights, and in one case, a bolt that had worked itself out and loosened the barrel).
If it can group a bunch of holes off to one side (or in your case, high/low), the shooter is doing their part and the sights need adjusting. If you have 20 shots all over the target but 4 or 5 seem to be a bit high/low, I'd suspect the shooter.