A nice, scholarly, article on the use of shotguns and laws of war from a U.S.-centric point of view:
http://www.lawofwar.org/Parks_Combat_Shotguns.htm
One thing you have to keep in mind is that when the Hague Conventions happened, it was A-OK to shred people to death with grapeshot. However, things like expanding bullets often left behind tiny fragments that were impossible to locate with the technology of the day and with no antibiotics, they were a source of infection as well.
As for shotguns, if they were effective in combat, they’d be used that way - laws of war or no. The fact fhat they are almost universally not used except in specialty roles - even by countries who do not observe the laws of war - is a big clue to their utility as a general use firearm.