Adapting the Turing Test for Declaring Document Analysis Problems Solved

Amazon's Mechanical Turk is a crowd sourcing technology that enables requesters to create tasks to be completed by human agents in exchange for compensation. Researchers in computer science have successfully used this service to quickly reach large numbers of subjects for a relatively low cost. However, the Mechanical Turk's model and policies introduce several experimental limitations and threats that must be controlled. In this short paper, we describe limitations imposed using Amazon's Mechanical Turk during an experiment on cyber-attack investigation techniques. While the experiment was successful, we were forced to change our experimental design and had to recover from some costly mistakes. The goal of this short paper is to identify these limitations and pitfalls and provide eight considerations for experimental design so that other researchers can maximize the benefits of using the Mechanical Turk as a research platform.

Crowdsourcing: A Critical Reflection on This New Frontier of Participant Recruiting in Nutrition and Dietetics Research

Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials

It is the pleasure of the organizing committee to welcome all participants to the 2010 IAPR Workshop on Document Analysis Systems (DAS). This year's workshop is being held June 9-11th in Boston, Massachusetts, a location situated in the heart of beautiful New England in the northeastern United States. Boston has a rich history that dates back to the 1600's and was a major focal point in the history of the American Revolution and America's fight for independence. Over the years, Boston has developed into a center for industrial, academic and cultural excellence and draws millions of visitors each year. DAS 2010 is the ninth workshop in a series. The first DAS was held in Kaiserslautern, Germany in 1994, and was followed by Malvern, PA (1996); Nagano, Japan (1998); Rio de Janeiro, Brazil (2000); Princeton, NJ (2002); Florence, Italy (2004); Nelson, New Zealand (2006) and Nara, Japan (2008). The DAS tradition is to bring together industry, academic and government researchers interested in many aspects of document analysis systems and to provide opportunities for fruitful interaction and collaboration. This year's workshop is organized as a three-day, single track event, with oral and poster presentations, as well as working group discussions on the second and third afternoons. Special sessions on contributed datasets and a keynote talk on this same topic provide a compelling theme, and we hope this focus will help push the field toward greater sharing of data and accepted standards for evaluation. This year, we received 91 submissions from 25 countries on six continents. The papers were reviewed by 45 members of our research community and an international program committee representing 16 different countries. The overall quality was excellent and we have chosen 28 full papers for oral presentation and 37 as poster papers, as well as 15 short papers that will be presented either as posters or in a special short oral format. In addition, six groups are scheduled to present live demos during the poster sessions. Full papers underwent the standard peer review process and will appear in the official workshop proceedings to be published in the ACM International Conference Proceedings Series, available online as part of the ACM Digital Library. The short papers are included in the unofficial hardcopy proceedings distributed at the event as well as on the DAS 2010 website.

The Macro Frames of Microwork: A Study of Indian Women Workers on AMT in the Post-Pandemic Moment

The smartFIX system, a commercial Document Analysis (DA) system, is introduced. It can be seen as a constraint solver over the domain of OCR alternatives. There are basically two types of constraints, database constraints and arithmetic constraints. The target of DA is then to optimize the constraint system, where the main challenge is the fuzziness of the OCR results. Efficiency is gained by a transition of the variables' state from 'soft' to 'hard'. Propagation is used to make sure that the result of the analysis is globally consistent. As an industrial application of (soft) constraints, smartFIX demonstrates the versatility of the generic constraints approach.

Document analysis and understanding (DAU) systems aim not only at the recognition of text but also at the extraction of relevant information out of a scanned document. Numerous studies have introduced efficient algorithms for document analysis, some of these studies proposed an articulated and/or a translation stage. Also, little work has been done in the English/Arabic (E/A) translation area. The main objective of this paper is to introduce a combination between the three trends (document understanding, E/A translation, and handling the output in an articulated voice). This paper focuses on the realisation of a bilingual articulated E/A system based on optical character recognition (OCR). The input of the proposed system will be an English or Arabic text through scanner or video camera; the output will be an articulated voice. The proposed scheme consists of two phases. The first phase consists of many processes, beginning with converting scanned document into an electronically processable form. Then, in the segmentation step, the essential problem in Arabic OCR, that is, how to cope with the various shapes of the same character, is solved. A new methodology for segmenting Arabic characters is presented. At the end of this phase, an efficient method of text recognition based on hybrid description (ANN, FFT) is used. In order to verify the performance of this phase, experiments with printed text were performed. The error rates were less than 0.1%. Results showed that the proposed scheme in this phase is very robust. In the second phase, a database with 3000 audio files was encapsulated to convert each word from the input text (the output of the first phase) to its correspondence. This research can help in many real-time applications such as immediate translation, a machine reader for blind people, and learning.

Amazon Mechanical Turk (MTurk) has been one of the most popular platforms for online research in psychology and the social sciences in general. While concerns about MTurk data quality have been raised, the platform continues to be widely used. The question is whether the MTurk platform is suitable for research and, if so, whether it is used optimally. We conducted a systematic investigation of MTurk data quality and reliability, including main and replication experiments, with more than 1300 participants subdivided into three cohorts: (i) workers (i.e. participants on the MTurk platform) with master requirement (i.e. high-performing workers selected by MTurk), (ii) workers without master requirement, and (iii) workers without master requirement, but with a 95% or above approval rate. We found that master workers almost never missed attentional checks, exhibited high reliability and showed no tendency towards straightlining, therefore, these workers are recommended, especially when the naivety of participants is not a strong prerequisite and no large sample size is required. In contrast, the workers without restrictions or with a 95% or above approval-rate threshold missed many attentional checks, exhibited low reliability and showed a tendency towards straightlining, raising serious concerns about the suitability of these workers for research.

Data collection using Internet-based samples has become increasingly popular in many social science disciplines, including advertising. This research examines whether one popular Internet data source, Amazon's Mechanical Turk (MTurk), is an appropriate substitute for other popular samples utilized in advertising research. Specifically, a five-sample between-subjects experiment was conducted to help researchers who utilize MTurk in advertising experiments understand the strengths and weaknesses of MTurk relative to student samples and professional panels. In comparisons across five samples, results show that the MTurk data outperformed panel data procured from two separate professional marketing research companies across various measures of data quality. The MTurk data were also compared to two different student samples, and results show the data were at least comparable in quality. While researchers may consider MTurk samples as a viable alternative to student samples when testing theory-driven outcomes, precautions should be taken to ensure the quality of data regardless of the source. Best practices for ensuring data quality are offered for advertising researchers who utilize MTurk for data collection.

Amazon’s Mechanical Turk (MTurk) is a relatively new website that contains the major elements required to conduct research: an integrated participant compensation system; a large participant pool; and a streamlined process of study design, participant recruitment, and data collection. In this article, we describe and evaluate the potential contributions of MTurk to psychology and other social sciences. Findings indicate that (a) MTurk participants are slightly more demographically diverse than are standard Internet samples and are significantly more diverse than typical American college samples; (b) participation is affected by compensation rate and task length, but participants can still be recruited rapidly and inexpensively; (c) realistic compensation rates do not affect data quality; and (d) the data obtained are at least as reliable as those obtained via traditional methods. Overall, MTurk can be used to obtain high-quality data inexpensively and rapidly.

In this commentary, we respond to Burnette et al.'s (2021) paper, which gives significant practical recommendations to improve data quality and validity while gathering data via Amazon's Mechanical Turk (MTurk). We argue that it is also important to acknowledge and review the specific ethical issues that might arise when recruiting MTurk workers as participants. We particularly raise three main ethical concerns that need to be addressed when recruiting research participants from participant recruitment platforms: participants' economic vulnerability, participants' sensitivity, and power dynamics between participants and researchers. We elaborate on these issues by discussing the ways in which they may appear and be responded to. We conclude that considering the ethical aspects of data collection and the potential impacts of data collection on those involved would complement Burnette et al.'s recommendations. Consequently, data collection processes should be transparent as well, in addition to data screening processes.

Behavioral economic frameworks emphasize the importance of contextual influences on alcohol use; therefore, identifying relative demand for alcohol versus other commodities is of importance. Cross-commodity purchase tasks allow participants to make choices across multiple concurrently available commodities and can thereby pinpoint interactions among those commodities. These tasks may help identify relevant substance-free alternative activities to target in alcohol treatment by determining whether the activity functions as a substitute for alcohol use. While substance-free activity promotion is a promising behavioral component of alcohol interventions, no research to-date has used behavioral economic methods to assess the substitutability of alternative activities for alcohol use. The present studies were preliminary assessments of novel single- and cross-commodity purchase tasks of various alternative activities (e.g., exercise, hobbies, civic involvement). Participants in Study 1 recruited from Amazon's Mechanical Turk (n = 110) were administered a series of novel activity purchase tasks and an alcohol purchase task. Results showed excellent fit of the exponential demand equation to activity purchase task data and provided initial support for adaptation of purchase task methodology to alternative activity demand. In Study 2, participants recruited from Amazon's Mechanical Turk (n = 108) were administered both single-commodity and cross-commodity purchase tasks of alcohol and exercise. While most participants demonstrate independent consumption of alcohol and exercise, a subset of participants substituted exercise for alcohol as indicated by quantitative cross-price elasticity indices. These response patterns highlight the importance of individual differences and hold implications for recovery efforts that promote alternative activity engagement and public policy. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Investigating aging Americans' transportation options in the era of crowdsourcing: Questionnaire design and survey implementation through Amazon Mechanical Turk

Over the last 30 years, legal scholars have increasingly deployed experimental studies, particularly hypothetical scenarios, to test intuitions about legal reasoning and behavior. That movement has accelerated in the last decade, facilitated in large part by cheap and convenient Internet participant recruiting platforms like Amazon Mechanical Turk. The widespread use of online subjects, a practice that dramatically lowers the barriers to entry for experimental research, has been controversial. At the same time, the field of experimental psychology is experiencing a public crisis of confidence widely discussed in terms of the “replication crisis.” At present, law and psychology research is arguably in a new era, in which it is both an accepted feature of the legal landscape and also a target of fresh skepticism. The moment is ripe for taking stock. In this article, we bring an empirical approach to these problems. Using three canonical law and psychology findings, we document the challenges and the feasibility of reproducing results across platforms. We evaluate the extent to which we are able to reproduce the original findings with contemporary subject pools (Amazon Mechanical Turk, other national online platforms, and in‐person labs). We partially replicate all three results, and show marked similarities in subject responses across platforms. In the context of the experiments here, we conclude that meaningful replication requires active intervention in order to keep the materials relevant and sensible. The second aim is to compare Amazon Mechanical Turk subjects to the original samples and to the replication samples. We find, consistent with the weight of recent evidence, that the Amazon Mechanical Turk samples are reasonably appropriate for these kinds of scenario studies. Subjects are highly similar to subjects on other online platforms and in‐person samples, though they differ in their high level of attentiveness. Finally, we review the growing replication literature across disciplines, as well as our firsthand experience, to propose a set of standard practices for the publication of results in law and psychology.

© 26th European Conference on Information Systems: Beyond Digitization - Facets of Socio-Technical Change, ECIS 2018. All Rights Reserved. Amazon Mechanical Turk (AMT) is a popular microtasking platform connecting those who need work done with those willing to do it. However AMT has come under increasing scrutiny for the way workers are treated on its platform. This paper examines one particular component of AMT's microtasking process by conducting a formal ethical analysis, using Tavani's Comprehensive Cyberethics Methodology, on the way work submitted for evaluation is assessed and either accepted or rejected. The study finds that the system is skewed in favour of those evaluating the work, with little recourse for workers, and an apparent disinterest from AMT. This paper contributes to the continuing debate over the governance of, and responsibilities for, those engaged in digital work through such platforms.