Paper reading #15: TurKit: human computation algorithms on mechanical turk

Reference Information:

Title: TurKit: Human Computation Algorithms on Mechanical Turk

Authors: Greg Little, Lydia B. Chilton, Max Goldman, Robert C, Miller

Conference: UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology

Summary:

Mechanical Turk (MTurk) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. Amazon uses such a mechanical turk in which users are paid small amounts of money for completing the human intelligence tasks (HITS) like writing product reviews. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit provides an API written in JavaScript for accomplishing MTurk Tasks. TurKit is a toolkit for prototyping and exploring algorithmic human computation, while maintaining a straight-forward imperative programming style. The authors present the crash-and-rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. The authors also present case studies of TurKit used for real experiments across different fields. The crash-and-rerun model favors usability over efficiency, but does so at an inherent cost in scalability. So, one of the major limitations of this system is expensive space complexity.

Discussion:

Companies like Amazon.com, ebay, other e-commerce companies and companies that rely on customer ratings have been using mechanical turks for quite some time now. They have to stick to this procedure since these tasks require human intelligence and can not be achieved using computer algorithms. These systems have a drawback when it comes to scalability, but with modern processors and cheap storage solutions, that doesn't quite seem to be an issue.

The name Mechanical Turk comes from "The Turk," a chess-playing automaton of the 18th century, which was made by Wolfgang von Kempelen