Figure from the paper

Aligning Superhuman AI with Human Behavior: Chess as a Model System

Reid McIlroy-Young, Russell Wang, Siddhartha Sen, Jon Kleinberg & Ashton Anderson

In Review 2020

Abstract pdf arΧiv

Even when machine learning systems surpass human ability in a domain, there are many reasons why AI systems that capture human-like behavior would be desirable: humans may want to learn from them, they may need to collaborate with them, or they may expect them to serve as partners in an extended interaction. Motivated by this goal of human-like AI systems, the problem of predicting human actions -- as opposed to predicting optimal actions -- has become an increasingly useful task. We extend this line of work by developing highly accurate personalized models of human behavior in the context of chess. Chess is a rich domain for exploring these questions, since it combines a set of appealing features: AI systems have achieved superhuman performance but still interact closely with human chess players both as opponents and preparation tools, and there is an enormous amount of recorded data on individual players. Starting with an open-source version of AlphaZero trained on a population of human players, we demonstrate that we can significantly improve prediction of a particular player's moves by applying a series of fine-tuning adjustments. Furthermore, we can accurately perform stylometry -- predicting who made a given set of actions -- indicating that our personalized models capture human decision-making at an individual level.

Figure from the paper

Aligning Superhuman AI with Human Behavior: Chess as a Model System

Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg & Ashton Anderson

KDD 2020

Abstract pdf arΧiv code Lichess

Press: WIRED Engadget TNW the morning paper agadmator U of T News

As artificial intelligence becomes increasingly intelligent---in some cases, achieving superhuman performance---there is growing potential for humans to learn from and collaborate with algorithms. However, the ways in which AI systems approach problems are often different from the ways people do, and thus may be uninterpretable and hard to learn from. A crucial step in bridging this gap between human and artificial intelligence is modeling the granular actions that constitute human behavior, rather than simply matching aggregate human performance. We pursue this goal in a model system with a long history in artificial intelligence: chess. The aggregate performance of a chess player unfolds as they make decisions over the course of a game. The hundreds of millions of games played online by players at every skill level form a rich source of data in which these decisions, and their exact context, are recorded in minute detail. Applying existing chess engines to this data, including an open-source implementation of AlphaZero, we find that they do not predict human moves well. We develop and introduce Maia, a customized version of Alpha-Zero trained on human chess games, that predicts human moves at a much higher accuracy than existing engines, and can achieve maximum accuracy when predicting decisions made by players at a specific skill level in a tuneable way. For a dual task of predicting whether a human will make a large mistake on the next move, we develop a deep neural network that significantly outperforms competitive baselines. Taken together, our results suggest that there is substantial promise in designing artificial intelligence systems with human collaboration in mind by first accurately modeling granular human decision-making.

Figure from the paper

From “Welcome New Gabbers" to the Pittsburgh Synagogue Shooting: The Evolution of Gab

Reid McIlroy-Young & Ashton Anderson

ICWSM 2019 (poster paper)

Abstract pdf arΧiv dataset

Gab, an online social media platform with very little content moderation, has recently come to prominence as an alt-right community and a haven for hate speech. We document the evolution of Gab since its inception until a Gab user carried out the most deadly attack on the Jewish community in US history. We investigate Gab language use, study how topics evolved over time, and find that the shooters' posts were among the most consistently anti-Semitic on Gab, but that hundreds of other users were even more extreme.

Figure from the paper

Interdisciplinarity and insularity in the diffusion of knowledge: an analysis of disciplinary boundaries between philosophy of science and the sciences

John McLevey, Alexander V. Graham, Reid McIlroy-Young, Pierson Browne & Kathryn S. Plaisance

Scientometrics 2018

Abstract link

Two fundamentally different perspectives on knowledge diffusion dominate debates about academic disciplines. On the one hand, critics of disciplinary research and education have argued that disciplines are isolated silos, within which specialists pursue inward-looking and increasingly narrow research agendas. On the other hand, critics of the silo argument have demonstrated that researchers constantly import and export ideas across disciplinary boundaries. These perspectives have different implications for how knowledge diffuses, how intellectuals gain and lose status within their disciplines, and how intellectual reputations evolve within and across disciplines. We argue that highly general claims about the nature of disciplinary boundaries are counterproductive, and that research on the nature of specific disciplinary boundaries is more useful. To that end, this paper uses a novel publication and citation network dataset and statistical models of citation networks to test hypotheses about the boundaries between philosophy of science and 11 disciplinary clusters. Specifically, we test hypotheses about whether engaging with and being cited by scientific communities outside philosophy of science has an impact on one’s position within philosophy of science. Our results suggest that philosophers of science produce interdisciplinary scholarship, but they tend not to cite work by other philosophers when it is published in journals outside of their discipline. Furthermore, net of other factors, receiving citations from other disciplines has no meaningful impact—positive or negative—on citations within philosophy of science. We conclude by considering this evidence for simultaneous interdisciplinarity and insularity in terms of scientific trading theory and other work on disciplinary boundaries and communication.

Figure from the paper

Introducing metaknowledge: Software for computational research in information science, network analysis, and science of science

John McLevey & Reid McIlroy-Young

Journal of Informetrics 2017

Abstract link code

metaknowledge is a full-featured Python package for computational research in information science, network analysis, and science of science. It is optimized to scale efficiently for analyzing very large datasets, and is designed to integrate well with reproducible and open research workflows. It currently accepts raw data from the Web of Science, Scopus, PubMed, ProQuest Dissertations and Theses, and select funding agencies. It processes these raw data inputs and outputs a variety of datasets for quantitative analysis, including time series methods, Standard and Multi Reference Publication Year Spectroscopy, computational text analysis (e.g. topic modeling, burst analysis), and network analysis (including multi-mode, multi-level, and longitudinal networks). This article motivates the use of metaknowledge and explains its design and core functionality.


Learning Personalized Models of Human Behaviour in Chess

Vector Institute Research Symposium (2021) slides

Maia Chess: A Human-Like Neural Network Chess Engine

Aavhan’s Online Lecture Series (2021) video slides

Maia Chess: A Human-Like Neural Network Chess Engine

ChessTech (2020) slides

Bridging the Gap between Superhuman AI and Human Behavior: Chess as a Model System

International Conference on Computational Social Science (2020) slides

Bridging the Gap Between Human and Artificial Intelligence in Chess

Evolution of Deep Learning Symposium (2019) poster

Generating and Analyzing Scientific Networks with Metaknowledge

1st North American Social Networks (NASN) Conference (2017)

metaknowledge: open source sofware for networks research on science

INSNA Sunbelt (2017)

How Knowledge Travels: An Analysis of the Diffusion of Philosophy of Science Over 60 Years

INSNA Sunbelt (2016)

The full BibTeX can be found here