AI for Social Science
Research Methods

Informal workplace networks—who interacts with whom, who shares information with whom, who mentors whom, who influences whom—serve as the fundamental mechanisms through which social closure operates, opportunities are hoarded, and group-based socioeconomic inequalities are reproduced. Yet real-time, large-sample data of such interactions are notoriously scarce, largely because private workspaces are, well, private. Yet occasionally, informal interpersonal connections in the workplace may become visible in public settings—at least among one group of professionals: academics! Academic conferences are key sites for building the kinds of professional networks that shape scholarly careers. Yet, it is hardly surprising that these informal interactions are often deeply segregated by subgroups such as race, gender, and seniority. In this study, we introduce large-scale image data and computer vision models as a novel approach to understanding information interaction segregation at academic conferences. First, we collect over 7,000 photographs of informal interaction groups at thirteen academic conferences organized by mainstream academic disciplines in the United States. Second, we identify interaction groups and code individual-level characteristics—gender, race, and age—in the images. The image annotation was carried out by both human coders and a locally fine-tuned open-source vision–language model. Finally, we compute various segregation metrics based on these group dynamics both within and across different conferences. Our findings challenge the notion that conferences are spaces where scholars simply "mingle by chance." Across conferences, informal interactions are strongly segregated along racial, gender, and age lines. The level of racial segregation in these face-to-face encounters is substantially higher than that observed in more stable and institutionalized settings such as residential neighborhoods, workplaces, or schools. This study also highlights the promise of harnessing new forms of data and methods to better understand "weak ties" in real-life settings.

Statistical decision algorithms are increasingly deployed in domains where ground-truth labels are hard to obtain, such as hiring, university admissions, and content moderation. In these settings, models are typically trained on historical human evaluations—for example, using past hiring decisions as a proxy for true applicant quality. However, if past evaluations unjustly favor certain groups, models trained on these labels may inherit those biases. To address this problem, we propose basing predictions on rubric embeddings, a representation framework that replaces standard black-box embeddings with features derived from expert-defined criteria that align with the underlying construct of interest. By anchoring predictions to semantically meaningful dimensions, this approach guards against biased proxy signals. We provide both theoretical and empirical evidence that rubric embeddings mitigate label bias under plausible conditions. Empirically, we evaluate our method on a novel dataset of applications to a large master's program. We find that models trained on rubric embeddings reduce group disparities while improving measures of cohort quality. Our results suggest that basing predictions on interpretable, domain-grounded representations offers a practical approach to learning in the presence of biased labels.

Generative AI creates a measurement problem for survey research. When respondents can outsource summarization, explanation, or fact retrieval, open-ended answers may no longer reflect their own reasoning. We introduce an auditable toolkit that combines response-process paradata with prompt-specific semantic benchmarks to detect AI use in open-ended responses. We validate the approach in a randomized survey experiment in which participants complete a summary task under either blocked AI access or observed access to an embedded AI tool, and by administering the same instrument to a synthetic AI agent. AI-assisted and automated responses leave distinct behavioral and textual traces: they show reduced drafting and revision, greater similarity to AI benchmark output, and lower distinctiveness relative to peer responses. These findings show that, in the age of generative AI, authorship must be measured rather than assumed.

We introduce GenAI-Powered Inference (GPI), a statistical framework for both causal and predictive inference using unstructured data, including text, image, and video. GPI leverages open-source Generative AI models—such as large language models and diffusion models—not only to generate unstructured data at scale but also to extract low-dimensional representations that capture their underlying structure. Applying machine learning to these representations, GPI enables estimation of causal and predictive effects while quantifying associated estimation uncertainty. Unlike existing approaches to representation learning, GPI does not require fine-tuning of generative models, making it computationally efficient and broadly accessible.

Observational studies of visual media effects are vulnerable to confounding bias due to the complexity of visual data, the difficulty in variable measurement, and the uncertainty of selection. We propose an inference framework that leverages multimodal LLMs and machine learning-based inference techniques to (1) identify potential confounders through reasoned visual assessment, (2) measure semantically complex features that are difficult to capture with conventional methods, and (3) apply double-Lasso for covariate selection and hierarchical OLS regression for effect estimation. We demonstrate this framework by revisiting the relationship between political personalization and social media engagement using 59,020 Instagram posts from U.S. politicians.

A dominant approach in quantitative social research is to represent data as a rectangle of numbers, where there is one row for each person and column for each variable about the people. This data representation is natural for survey data and fits well with existing methods, such as the GLM and supervised machine learning. In this talk, we begin to explore an alternative pipeline that represents data about people as a text "book of life" and then analyzes the data using LLMs. This approach creates affordances that have no obvious analogue in existing approaches, and may be especially valuable for life course data with complex temporal, network, and hierarchical structure.

We study the book of life + LLM pipeline in two distinct settings focused on predicting life outcomes: one complex (the Dutch population registry) and one simple (the US American Community Survey). We compare approaches empirically and try to isolate the source of performance differences. The talk concludes with a brief discussion of the shared infrastructure—community-determined "model organisms" and open-source software—that might help us discover the value (or lack thereof) of the new approach more quickly and reliably.

Joint work with Sarah Pedersen, Mattie Niznik, Stephan Rabanser, Varun Satish, Flavio Hafner, Sayash Kapoor, Malte Luken, Lydia T. Liu, Tiffany Liu, Juan C. Perdomo, Benedikt Stroebl, Keyon Vafa, and Mark Verhagen.

Large language models (LLMs) have prompted proposals to replace human subjects in social science experiments with simulated responses. Empirical evaluations suggest that this practice—often called silicon sampling—can sometimes approximate human behavior but is unreliable. We delineate where this approach may still provide value and where it may not, but primarily study an alternative approach: one in which model-based predictions are used not as substitutes for human data, but as auxiliary measurements within randomized experiments. We formalize the inference of causal estimands from mixed-subjects randomized controlled trials, in which outcomes are observed for a subset of units while predictions are available for all units. Under transparent design conditions, we derive a family of estimators that remain unbiased for the average treatment effect in finite samples while exploiting predictions to reduce variance. We characterize when prediction-powered, calibration-based, arm-specifically tuned, and difference-in-predictions estimators improve precision, and we provide a software package which operationalizes these results and aids researchers to jointly select estimators and allocate budgets between human data collection and prediction generation. Together, our results show how generative artificial intelligence can improve experimental social science without compromising scientific validity.

Social media research increasingly studies environments rather than static stimuli: feeds evolve, users interact, and generative AI can now populate platforms with synthetic actors that respond dynamically. We describe the Social Media Accelerator as research infrastructure for this setting: a controlled platform for randomized social-media experiments with embedded surveys, synthetic users, behavioral logs, and auditable exposure records. The talk then considers how GenAI enters the scientific workflow itself, including literature synthesis, coding, measurement, drafting, and validation. Prompts, model versions, generated outputs, coding decisions, validation checks, and human revisions can be treated as part of the research record. The broader aim is to support accountable AI-assisted social science in a world where both social behavior and the work of studying it increasingly involve AI.

Millions of people around the world query large language models for information. While several studies have compellingly documented the persuasive potential of these models, there is limited evidence of who or what influences the models themselves, leading to a flurry of concerns about which companies and governments build and regulate the models. We show through six studies that government control of the media across the world already influences the output of large language models (LLMs) via their training data.

We use a cross-national audit to show that LLMs exhibit a stronger pro-government valence in the languages of countries with lower media freedom than those with higher media freedom. This result is correlational so to triangulate the specific mechanism of how state media control can influence LLMs, we develop a multi-part case study on China's media. We demonstrate that media scripted and curated by the Chinese state appears in large language model training datasets. To evaluate the plausible effect of this inclusion, we use an open-weight model to show that additional pretraining on Chinese state-coordinated media generates more positive answers to prompts about Chinese political institutions and leaders.

We link this phenomenon to commercial models through two audit studies demonstrating that prompting models in Chinese generates more positive responses about China's institutions and leaders than do the same queries in English. The combination of influence and persuasive potential across languages suggests the troubling conclusion that states and powerful institutions have increased strategic incentives to leverage media control in the hopes of shaping large language model output.

Computational reproducibility is central to scientific credibility, yet verifying published results at scale remains costly. We develop an AI-assisted workflow for automated full-paper replication—retrieving materials, reconstructing environments, executing code, and matching outputs to point estimates reported in regression tables. We define a universe of all empirical and quantitative papers from the three top political science journals (2010–2025) and measure stated data availability using automated extraction. For a stratified sample of 384 studies, we apply the workflow to conduct full-paper replication, totaling 3,382 empirical models. We find that journal verification requirements, combined with data archiving mandates, drive reproducibility: the full-paper reproducibility rate rises significantly due to DA-RT, and conditional on accessible replication packages, the vast majority of papers are fully or largely reproducible. As a secondary application, we apply standardized IV diagnostics to 92 studies (215 specifications), illustrating how automated execution enables systematic reanalysis across heterogeneous empirical settings.

Large Language Models (LLMs) are increasingly used in social science research to annotate unstructured data, often replacing research assistants and experts. Yet using these predicted annotations in downstream statistical analyses can yield biased estimates—a problem compounded by the black-box and stochastic nature of LLMs. This study evaluates the consequences of LLM annotation for empirical political science research. We conduct a systematic replication and reanalysis of 14 recently published papers from leading political science journals, re-annotating data originally coded by humans or supervised models with 15 different open-weight and proprietary LLMs. Analyzing hundreds of millions of annotations, we find that LLM annotations have low intercoder reliability with the original annotations and moderate reliability among the LLMs themselves. Smaller and reasoning models are particularly sensitive to minor variations in artifacts such as prompt design. As a result, downstream estimates derived from different sets of annotations show significant variation, altering conclusions in more than one-third of cases. Mitigation strategies, such as in-context learning and bias correction techniques, are useful but have limitations. Based on these findings, we propose best practices for using LLMs for annotation and provide an open-source R package, localLLM, to support their implementation.

Images are increasingly used as stimuli in social science experiments, but image construction introduces two design challenges that text vignettes avoid. First, the image may fail to depict the intended treatment attributes clearly (construct validity). Second, because images carry a complete scene, they inevitably include non-treatment features that may differ across conditions and confound the treatment comparison (internal validity). We distinguish uncontrolled construction, where non-treatment features are left to the source or the model, from controlled construction, where the researcher deliberately decides which features vary and which stay fixed. We propose two controlled approaches: decomposed AI generation, which builds stimuli from scratch by generating reusable components, and image editing, which modifies only the treatment-relevant elements of an existing image. We develop measures of construct validity and unconfoundedness and evaluate these approaches in two empirical settings. Study 1 compares natural sourcing, simple AI generation, and decomposed AI generation on a 216-condition factorial design. Decomposed AI generation achieves the highest construct validity and lowest confounding. Study 2 compares manual and AI image editing on a 75-condition conjoint extension of Butler and Tavits (2017). Manual editing achieves near-zero confounding but at substantially higher effort than AI editing.

Who speaks in local democracy, and how much? Despite growing scholarly interest in political participation beyond voting, we lack scalable tools to measure citizen voice in local government. We develop an automated audio analysis pipeline that classifies speakers in city council meetings as elected officials or members of the public—directly from audio, using neural speaker diarization combined with multimodal LLM few-shot classification.

We introduce two methodological innovations: a temporal exemplar selection with iterative bootstrapping strategy, and a multi-agent deliberation framework. An AND ensemble combining the baseline and bootstrapped classifiers achieves 93.2% accuracy (κ = 0.79), citizen precision of 0.82, and citizen F1 of 0.83—substantially outperforming any single method. Our pipeline targets 1,600+ meetings from seven U.S. rent control cities and their neighbors (2017–2023).