CMU Crowdsourcing Lunch Seminar: Panos Ipeirotis

We describe Quizz, a gamified crowdsourcing system that simultaneously

assesses the knowledge of users and acquires new knowledge from them.

Quizz operates by asking users to complete short quizzes on specific

topics; as a user answers the quiz questions, Quizz estimates the

user’s competence. To acquire new knowledge, Quizz also incorporates

questions for which we do not have a known answer; the answers given

by competent users provide useful signals for selecting the correct

answers for these questions. Quizz actively tries to identify

knowledgeable users on the Internet by running advertising campaigns,

effectively leveraging “for free” the targeting capabilities of

existing, publicly available, ad placement services. Quizz quantifies

the contributions of the users using information theory and sends

feedback to the advertising system about each user. The feedback

allows the ad targeting mechanism to further optimize ad placement.

Our experiments, which involve over ten thousand users, confirm that

we can crowdsource knowledge curation for niche and specialized

topics, as the advertising network can automatically identify users

with the desired expertise and interest in the given topic. We present

controlled experiments that examine the effect of various incentive

mechanisms, highlighting the need for having short-term rewards as

goals, which incentivize the users to contribute. Finally, our

cost-quality analysis indicates that the cost of our approach is below

that of hiring workers through paid-crowdsourcing platforms, while

offering the additional advantage of giving access to billions of

potential users all over the planet, and being able to reach users

with specialized expertise that is not typically available through

existing labor marketplaces.

Bio:

Panos Ipeirotis is a Professor and George A. Kellner Faculty Fellow at

the Department of Information, Operations, and Management Sciences at

Leonard N. Stern School of Business of New York University. He

received his Ph.D. degree in Computer Science from Columbia University

in 2004. He has received nine “Best Paper” awards and nominations and

is the recipient of the 2015 Lagrange Prize, for his contributions in the

field of social media, user-generated content, and crowdsourcing.