After some further searching, It appears that while we have some logic in 4th and 5th amendment questions on this privacy issue, the answer to juror privacy lies in the first and fourteenth amendment. (Yeah, aside from the 14th being one big giant catch-all I don't know)
Brandborg v Lucas Unfortunately it's not a government website, and I can't be sure those are the actual words of the court reprinted so due to possible copyright issues, I won't quote anything.
Basics are: A lady got summoned. Had a questionnaire. Refused to answer 12 questions (and inadvertently overlooked a thirteenth). She went to her court date with her partially filled out questionnaire. The judge ordered her to finish it. She refused. She was taken out of court, the Judge and the lawyers talked, with both the prosecution and defense somewhere between suggesting and asking she be held in contempt. (Part of why I think it might not be a direct quote of the decision, as I can't imagine that sort of thing in the official record) She was held in contempt for a 3 day jail sentence, and a $200 fine. She appealed. Her contempt citation was set aside.