Several things here do not quite match.
Flintlocks were 20 years obsolete by 1864, the Army had been issuing caplocks since 1842.
A C.W. era rifle, rifle musket, or rifled musket (not the same thing) would have had a rear sight. Might have been lost off but I would expect the location to still be visible as a flat or dovetail.
Pictures would be a big help to identify.