Given two texts, we may ask which one is more perusasive. Such a comparison only tells us about these two texts and does not tell us what elements of the text drive the causal mechanism. Since the mechanism is of interest, a tempting design is to show many texts, measure their effects, and use natural language processing to learn what features of the texts should be considered as components of a causal analysis. However, such a black box approach (e.g. a large language model) provides insufficient control of the causal model and may lead to spurious or nonsensical results.

We develop a novel experimental design for text as treatment that controls which elements of text are being studied and admits simple estimators. We study the effect of intellectually humble language on the persuasiveness of text. Using our data we demonstrate two major issues with existing machine learning tools for inferring causal effects of text. Transformer models that use learned representations of text as confounders overfit the data, inducing positivity violations. Other tools that try and correct for text indirectly underfit the data and act like estimators that never even looked at text confounders.