Using Contextual Embeddings to Predict the Effectiveness of Novel Heterogeneous Treatments
ABSTRACT: Our study demonstrates the power of contextual embeddings for predicting the performance of novel heterogeneous treatments. Our proposed framework leverages four key benefits of machine learning: prediction, enhanced causal estimation, estimation of heterogeneous effects, and generative capability. To test our framework, we exploit a targeted marketing setting in which 34 email promotions were sent to 1.3 million customers over a 45-day period. Using these emails as treatments, we start by estimating the doubly robust scores of customer-level purchase amounts to serve as our target variable. We incorporate customer-level demographics and contextual embeddings, which capture the context of the latent states, to estimate the response function of these emails. Using a series of leave-one-out exercises, we show how our approach can accurately extrapolate the average performance, heterogeneous performance, and recommended targeting policies of novel promotions. We find that our framework recovers 78.6% of the variation of the aggregate treatment effects, an average of 65.36% of the variation in the heterogeneous treatment effects of each novel treatment, and matches 82% of policy recommendations made using the true signals. We conclude our study with an example of leveraging Generative AI to create novel treatments and then evaluate their performance with our framework.
Dr. Paul Ellickson - Bio
Professor Ellickson’s research interests lie at the intersection between quantitative marketing and industrial organization, with a focus on using structural modeling to understand the forces that drive strategic interaction and optimal decision making. He is particularly interested in modeling the importance of dynamic and spatial competition in retail trade. Ellickson’s research has been published in various academic journals including the Review of Economic Studies, the RAND Journal of Economics, Marketing Science, the Journal of Marketing Research, Quantitative Marketing and Economics, and the Journal of Economic Perspectives.
REGISTRATION CLOSED