It has been done. A friend has one of those Single Six to .38 Special conversions by John Gallagher and it is a neat little gun. No reason not to make up a .32.
It won't be cheap and it won't be fast.
You finding a barrel and cylinder would help but I think the work would still cost more than a factory gun originally in the caliber.
There is a Vaquerito = .32 with hogwallow fixed sights, on Gunbroker with a "starting bid" of $429 and a buy it now of $599.
A Super Single Six pattern .32 with adjustable sights with an opening bid of $649.95, b.i.n. of $749.95.
No actual bids in on either, yet.
That may sound high, but a talk with a custom gunsmith will show you what a high price is.