Whose voices - whose labels - should a machine learning algorithm learn to emulate? For AI tasks ranging from online comment toxicity detection to poster design to medical treatment, different groups in society may have irreconcilable disagreements about what constitutes ground truth. Today's supervised machine learning pipeline typically resolves these disagreements implicitly by majority vote over annotators' opinions. This majoritarian procedure abstracts individual people out of the pipeline and collapses their labels into an aggregate pseudo-human, ignoring minority groups' labels. In this talk, I will present Jury Learning: an interactive AI architecture that enables developers to explicitly reason over whose voice a model ought to emulate through the metaphor of a jury. Through our exploratory interface, practitioners can declaratively define which people or groups, in what proportion, determine the classifier's prediction. To evaluate models under societal disagreement, I will also present The Disagreement Deconvolution: a metric transformation showing how, in abstracting away the individual people that models impact, current metrics dramatically overstate the performance of many user-facing tasks. These components become building blocks of a new pipeline for encoding our goals and values in human-AI systems, which strives to bridge principles of HCI with the realities of machine learning.