Rating Worker Skills and Task Strains in Collaborative Crowd Computing: A Competitive Perspective

In micro-task crowdsourcing markets such as Amazon's Mechanical Turk, how to obtain high quality result without exceeding the limited budgets is one main challenge. The existing theory and practice of crowdsourcing suggests that uneven task difficulty plays a crucial role to task quality. Yet, it lacks a clear identifying method to task difficulty, which hinders effective and efficient execution of micro-task crowdsourcing. This paper explores the notion of task difficulty and its influence to crowdsourcing, and presents a difficulty-based crowdsourcing method to optimize the crowdsourcing process. We firstly identify task difficulty feature based on a local estimation method in the real crowdsourcing context, followed by proposing an optimization method to improve the accuracy of results, while reducing the overall cost. We conduct a series of experimental studies to evaluate our method, which show that our difficulty-based crowdsourcing method can accurately identify the task difficulty feature, improve the quality of task performance and reduce the cost significantly, and thus demonstrate the effectiveness of task difficulty as task modeling property.

Amazon's Mechanical Turk is a crowd sourcing technology that enables requesters to create tasks to be completed by human agents in exchange for compensation. Researchers in computer science have successfully used this service to quickly reach large numbers of subjects for a relatively low cost. However, the Mechanical Turk's model and policies introduce several experimental limitations and threats that must be controlled. In this short paper, we describe limitations imposed using Amazon's Mechanical Turk during an experiment on cyber-attack investigation techniques. While the experiment was successful, we were forced to change our experimental design and had to recover from some costly mistakes. The goal of this short paper is to identify these limitations and pitfalls and provide eight considerations for experimental design so that other researchers can maximize the benefits of using the Mechanical Turk as a research platform.

Crowdsourcing: A Critical Reflection on This New Frontier of Participant Recruiting in Nutrition and Dietetics Research

Crowdsourcing markets such as Amazon's Mechanical Turk provide an enormous potential for accomplishing work by combining human and machine computation. Today crowdsourcing is mostly used for massive parallel information processing for a variety of tasks such as image labeling. However, as we move to more sophisticated problem-solving there is little knowledge about managing dependencies between steps and a lack of tools for doing so. As the contribution of this paper, we present a concept of an executable, model-based programming language and a general purpose framework for accomplishing more sophisticated problems. Our approach is inspired by coordination theory and an analysis of emergent collective intelligence. We illustrate the applicability of our proposed language by combining machine and human computation based on existing interaction patterns for several general computation problems.

Comparing Amazon Mechanical Turk with unpaid internet resources in online clinical trials

Amazon’s Mechanical Turk is an online service that connects people all over the world who are willing to work for monetary compensation with companies and individuals who need simple tasks completed that are best done by humans. This paper examines the benefits and caveats from the use of Mechanical Turk as a platform to conduct an online usability test. Our findings indicate that Mechanical Turk can be a very useful tool when it is employed to collect basic user generated data such as click time and location, text input, and subjective usability data. In situations where more complex usage data is required, Mechanical Turk can still be a valuable resource to run a pilot test to assist in determining potential usability problems. However, in its current form, Mechanical Turk has limitations and can only be used to assess websites; it cannot be relied upon to conduct a usability test on software without the use of a virtual machine.

Data collection using Internet-based samples has become increasingly popular in many social science disciplines, including advertising. This research examines whether one popular Internet data source, Amazon's Mechanical Turk (MTurk), is an appropriate substitute for other popular samples utilized in advertising research. Specifically, a five-sample between-subjects experiment was conducted to help researchers who utilize MTurk in advertising experiments understand the strengths and weaknesses of MTurk relative to student samples and professional panels. In comparisons across five samples, results show that the MTurk data outperformed panel data procured from two separate professional marketing research companies across various measures of data quality. The MTurk data were also compared to two different student samples, and results show the data were at least comparable in quality. While researchers may consider MTurk samples as a viable alternative to student samples when testing theory-driven outcomes, precautions should be taken to ensure the quality of data regardless of the source. Best practices for ensuring data quality are offered for advertising researchers who utilize MTurk for data collection.

Amazon’s Mechanical Turk (MTurk) is a relatively new website that contains the major elements required to conduct research: an integrated participant compensation system; a large participant pool; and a streamlined process of study design, participant recruitment, and data collection. In this article, we describe and evaluate the potential contributions of MTurk to psychology and other social sciences. Findings indicate that (a) MTurk participants are slightly more demographically diverse than are standard Internet samples and are significantly more diverse than typical American college samples; (b) participation is affected by compensation rate and task length, but participants can still be recruited rapidly and inexpensively; (c) realistic compensation rates do not affect data quality; and (d) the data obtained are at least as reliable as those obtained via traditional methods. Overall, MTurk can be used to obtain high-quality data inexpensively and rapidly.

In this commentary, we respond to Burnette et al.'s (2021) paper, which gives significant practical recommendations to improve data quality and validity while gathering data via Amazon's Mechanical Turk (MTurk). We argue that it is also important to acknowledge and review the specific ethical issues that might arise when recruiting MTurk workers as participants. We particularly raise three main ethical concerns that need to be addressed when recruiting research participants from participant recruitment platforms: participants' economic vulnerability, participants' sensitivity, and power dynamics between participants and researchers. We elaborate on these issues by discussing the ways in which they may appear and be responded to. We conclude that considering the ethical aspects of data collection and the potential impacts of data collection on those involved would complement Burnette et al.'s recommendations. Consequently, data collection processes should be transparent as well, in addition to data screening processes.

Behavioral economic frameworks emphasize the importance of contextual influences on alcohol use; therefore, identifying relative demand for alcohol versus other commodities is of importance. Cross-commodity purchase tasks allow participants to make choices across multiple concurrently available commodities and can thereby pinpoint interactions among those commodities. These tasks may help identify relevant substance-free alternative activities to target in alcohol treatment by determining whether the activity functions as a substitute for alcohol use. While substance-free activity promotion is a promising behavioral component of alcohol interventions, no research to-date has used behavioral economic methods to assess the substitutability of alternative activities for alcohol use. The present studies were preliminary assessments of novel single- and cross-commodity purchase tasks of various alternative activities (e.g., exercise, hobbies, civic involvement). Participants in Study 1 recruited from Amazon's Mechanical Turk (n = 110) were administered a series of novel activity purchase tasks and an alcohol purchase task. Results showed excellent fit of the exponential demand equation to activity purchase task data and provided initial support for adaptation of purchase task methodology to alternative activity demand. In Study 2, participants recruited from Amazon's Mechanical Turk (n = 108) were administered both single-commodity and cross-commodity purchase tasks of alcohol and exercise. While most participants demonstrate independent consumption of alcohol and exercise, a subset of participants substituted exercise for alcohol as indicated by quantitative cross-price elasticity indices. These response patterns highlight the importance of individual differences and hold implications for recovery efforts that promote alternative activity engagement and public policy. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

In various human-agent collaborative settings the agent is fault-prone. In many of these settings, it is possible that the human user will also account to failure, hindering task execution. In this paper we study the effect of the latter type of failures over the user’s satisfaction with the agent in the collaborative setting. We report the results of two experiments, differing in the number of agent’s faults and the way faults influence task progress and attract players’ focus, with 264 subjects recruited and interacted through Amazon Mechanical Turk. We find that when the user accounts for some faults during the collaborative execution of the task, she becomes more forgiving to the agent faults, and consequently more satisfied with the collaboration, compared to the case where she makes no faults. The importance of this finding becomes most apparent in the design of collaborative agents.

We explore how microwork platforms manage difficult tasks in paid crowdsourcing environments. We argue that as human computation becomes more prevalent, notably in the context of big data ecosystems, microwork platforms might have to evolve and to take a more managerial stance in order to provide the right incentives to online workers to handle difficult tasks. We illustrate this first through a name disambiguation experiment on Amazon Mechanical Turk (AMT), a well-known microwork platform, and second through direct analysis of the dynamics of task execution in a dataset of real microwork projects on AMT. We discuss the emergence of more specialised microwork platforms as an attempt to facilitate a better management of difficult tasks in the context of paid crowdsourcing.

Deep Gaussian processes (DGPs) are the natural extension of Gaussian processes (GPs) to a multi-layer architecture. DGPs are powerful probabilistic models that have shown better results than standard GPs in terms of generalization performance and prediction uncertainty estimation. Nevertheless, exact inference in DGPs is intractable, making these models hard to train. For this task, current approaches in the literature use approximate inference techniques such as variational inference or approximate expectation propagation. In this work, we present a new method for inference in DGPs using an approximate inference technique based on Monte Carlo methods and the expectation propagation algorithm. Our experiments show that our method scales well to large datasets and that its performance is comparable or better than other state of the art methods. Furthermore, our training method leads to interesting properties in the predictive distribution of the DGP. More precisely, it is able to capture output noise that is dependent on the input and it can generate multimodal predictive distributions. These two properties, which are not shared by other state-of-the-art approximate inference methods for DGPs, are analyzed in detail in our experiments.

This dissertation consists of three studies that investigate the information privacy & security associated with healthcare technology use. Essay 1 PRISMA-style systematically reviews the existing literature on privacy information disclosure in IoT technology and serves as the theoretical foundation of the current research. It is crucial to comprehend why, how, and under what consequences individuals choose to disclose their personal and health information since doing so is beneficial to the company. This SLR method allows us to find those factors that significantly impact individuals' behavioral intention to disclose personal information while using IoT technologies. Essay 2 posits, develops, and tests a comprehensive theoretical framework built upon the theory of planned behavior and the health belief model to examine factors affecting willingness to disclose PHI in order to use WFDs. A research survey is designed and distributed to a crowdsourcing platform, Mechanical Turk (M-Turk). Research hypotheses are tested using partial least square – structural equation modeling (PLS-SEM). To achieve this purpose, Essay 3 extends the findings from the previous essay and further investigates the caregiver context. Therefore, we developed a novel theoretical model utilizing privacy calculus theory and the technology acceptance model to investigate the willingness of the elderly to disclose personal health information needed to use caregiver robots. Survey data were collected using crowdsourcing utilizing Amazon's Mechanical Turk (M-Turk) and Prolific. Research hypotheses are tested using partial least square – structural equation modelling (PLS-SEM). The findings provide value for academia, practitioners, and policymakers.

This paper describes how a set of stories, each conveying a personality trait from the Five Factor Model at a high or low level, were developed using Amazon's Mechanical Turk. These stories will be used to develop interfaces that adapt to personality using a User as Wizard method. The paper shows how difficult it is to construct stories that convey a single personality trait. It also shows how such stories can be constructed for most cases, and how Mechanical Turk can aid to achieve this.