New Experimental Designs With Generative AI — From Texts, To Networks, To Interacting Agents

Given two texts, we may ask which one is more persuasive. Such a comparison only tells us about these two texts and does not tell us what elements of the text drive the causal mechanism. Since the mechanism is of interest, a tempting design is to show many texts, measure their effects, and use natural language processing to learn what features of the texts should be considered as components of a causal analysis. However, such a black box approach (e.g. a large language model) provides insufficient control of the causal model and may lead to spurious or nonsensical results.

In this talk we develop a novel experimental design for text as treatment that controls which elements of text are being studied and admits simple estimators. Using data from a randomized experiment we then demonstrate two major issues with existing machine learning tools for inferring causal effects of text. Transformer models that use learned representations of text as confounders overfit the data, inducing positivity violations. Other tools that try and correct for text indirectly underfit the data and act like estimators that never even looked at text confounders.

Nevertheless, Generative AI technologies are redefining the very questions that researchers in political science, sociology, and health science can address. These cutting-edge tools, which are increasingly capable of producing coherent texts, modeling complex networks, and simulating interacting agents, are gradually upending traditional research paradigms. As AI blurs the boundaries between observation and intervention, scholars are forced to rethink their experimental designs and the underlying assumptions about human behavior and social structures. There is an urgent need for entirely new research environments—platforms that can harness the potential of generative AI to perform controlled, high-throughput experiments. We developed the Human-AI Research Platform, a platform that creates virtual social environments where researchers can systematically manipulate features, deploy automated agents, and observe emergent behaviors in a controlled manner. In this talk we will demonstrate the platform, as well as some of the experiments we have run on it thus far.

To the worried audience-member that realizes this is too much for one talk: At the end of the day, the talk is about the difficulties associated with causal identification in modern experimental designs and provides several directions forward.