Artificial intelligence is rapidly transforming political science research, enabling tasks that once took years to be completed in hours, but leading scholars warn the field is moving faster than its ability to verify what these tools measure.

These were the central themes of a lecture by Prof. Sarah Brooks of Ohio State University, delivered at the 5th Summer Training on Qualitative and Quantitative Methods (STQ2M), hosted by the Faculty of Social Sciences. Her lecture, entitled Politics by the Numbers: AI and the Transformation of Political Science Research, offered a wide-ranging assessment of how large language models are reshaping the discipline, and at what cost.

Prof. Brooks outlined how AI is already being used across political science: classifying party manifestos across multiple languages, scoring legislative speeches for ideological position, and annotating thousands of political texts at a fraction of the cost of traditional human coding. A landmark study found that AI-based annotation outperformed crowdsourced human coders by roughly 25 percentage points in accuracy, at around 30 times lower cost.

“AI is unlocking a lot opportunities for more efficient and large data gathering. However, it also presents new challenges for political science research, such as identifying biases, addressing opacity, and confronting the difficulties and inequalities that arise. The use of AI by some scholars to produce research quickly creates a disparity compared to those who do not use it,” she said.

But the more provocative development she highlighted is the use of AI to simulate human survey respondents — so-called “silicon samples.” Researchers have experimented with conditioning language models on detailed demographic profiles and asking them to answer survey questions as those people would. The appeal is clear at a time when polling response rates are collapsing and probability sampling is growing prohibitively expensive.

The results, however, are inconsistent. While AI respondents perform reasonably well on neutral questions, they systematically diverge from real humans on sensitive topics involving race, gender, and immigration — skewing more progressive than the populations they claim to represent. Responses also shift unpredictably when prompts are slightly reworded, and change over time as underlying models are quietly updated by their developers.

“Research on a large language model is not research on a population,” one survey firm noted. “It’s research on a model of the population.”

Prof. Brooks also raised concerns about reproducibility, since AI outputs can differ even when the same prompt is used twice. Bias embedded in commercial models poses a further methodological risk, as does the concentration of AI infrastructure in the hands of a small number of private companies — OpenAI, Anthropic, Google, and Meta — who can alter their systems without notice.

Prof. Brooks sees that AI will not replace political science. She ended on a note of cautious optimism. AI tools are real, powerful, and getting more powerful. Ignoring them would be a mistake. But so would trusting them blindly.

The discipline’s job, she argued, is to ask the same hard question of AI-generated findings that it asks of any other research: What exactly have we learned here — and how confident should we be?