Technology has to advance on a broad front, as mentioned here already how ammunition technology was probably the biggest limiting factor at that time. Other things would be addressed as the problems appeared with newer technologies, such as high velocity cartridges that came along 20 years later. But mostly, it was a matter of simply thinking of something. Everyone has a moment when they say "why didn't I think of that?"
A lot of work and development goes into something that eventually becomes a dead end, at least for military purposes. Think of tube magazines.
There were successful repeating rifles during the Civil War and much better ones not much later. They used tube magazines. Tube magazines are still used, of course, but not in the army. Someone had to think of a different way to carry the ammuntion in the weapon. Box magazines were the thing by 1890 but it took a war for people to realize that loading one at a time was suddenly very old-fashioned. British soldiers did carry a spare magazine for a while before they had "charger loading," although it isn't clear to me why that was done. In any case, the magazine was still considered to be an emergency reserve, an idea that is not quite dead yet.
Military organizations tend to be conservative, you know, and there is often a built-in resistance to new things. That was just as true then as it is now. And yet another factor that we often overlook is that sometimes the wonderful advantages we take for granted in something new had serious shortcomings at the time. In the case of small arms, the use of black powder caused problems already mentioned. In the case of artillery, smoothbore artillery remained in use during the Civil War because it worked well and in some ways better than rifled muzzle loaders that were also widely used. Some of the artillery rifles also had an unfortunate tendency to blow up at the wrong time.
Shoot low, sheriff. They're riding Shetlands!
Underneath the starry flag, civilize 'em with a Krag,
and return us to our own beloved homes!
Buy War Bonds.