No matter where the lead comes from and what it has in it, lead itself is poisonous; contact with it or breathing the fumes from molten lead can result in something unexpected - your early (and painful) death.
Please do bullet casting and reloading with lead bullets in a well ventilated area, handle lead with rubber or latex gloves, and avoid the fumes from the melting pot.
I've been casting bullets since 1972 omg, that's 41 years...anyway I'm pretty sure you are safe from lead vaper @ 700 farenheit and if you aren't a thumb sucker you don't need gloves if you wash your hands after a casting session. I know of one casting death, a pot full of unknown 'stuff', probably from batteries or other nasties.
Handling lead or lead alloys is not a death defyin process. Common sense works.