WO2013003861A3 - Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks - Google Patents

Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks Download PDF

Info

Publication number
WO2013003861A3
WO2013003861A3 PCT/US2012/045321 US2012045321W WO2013003861A3 WO 2013003861 A3 WO2013003861 A3 WO 2013003861A3 US 2012045321 W US2012045321 W US 2012045321W WO 2013003861 A3 WO2013003861 A3 WO 2013003861A3
Authority
WO
WIPO (PCT)
Prior art keywords
job
tasks
worker
evaluating
providing
Prior art date
Application number
PCT/US2012/045321
Other languages
French (fr)
Other versions
WO2013003861A2 (en
Inventor
David OLESON
Alexander Sorokin
Greg LAUGHLIN
Vaughn HESTER
John LE
Christopher R. Van Pelt
Lukas A. Biewald
Original Assignee
CrowdFlower, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US13/275,290 external-priority patent/US8554605B2/en
Application filed by CrowdFlower, Inc. filed Critical CrowdFlower, Inc.
Publication of WO2013003861A2 publication Critical patent/WO2013003861A2/en
Publication of WO2013003861A3 publication Critical patent/WO2013003861A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06398—Performance of employee with respect to a job function

Abstract

Systems and methods for evaluating a worker in performing crowd sourced tasks and providing in-task training are disclosed. In one aspect, embodiments of the present disclosure include a method, which may be implemented on a system, for selecting a job distributed through a job distribution platform for workers to work on, for use to generate a test task, the job being associated with a known correct result, associating a manipulated result, known to be an incorrect result for the job, with the job to generate the test task, and/or presenting the job with the manipulated result as the test task to a worker for evaluation of work quality of the worker. The job distribution platform crowd sources tasks online to workers to work on via their respective computing devices.
PCT/US2012/045321 2011-06-30 2012-07-02 Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks WO2013003861A2 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201161503483P 2011-06-30 2011-06-30
US61/503,483 2011-06-30
US13/275,290 US8554605B2 (en) 2011-06-29 2011-10-17 Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks
US13/275,290 2011-10-17

Publications (2)

Publication Number Publication Date
WO2013003861A2 WO2013003861A2 (en) 2013-01-03
WO2013003861A3 true WO2013003861A3 (en) 2013-03-14

Family

ID=47424833

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2012/045321 WO2013003861A2 (en) 2011-06-30 2012-07-02 Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks

Country Status (1)

Country Link
WO (1) WO2013003861A2 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10726377B2 (en) * 2015-12-29 2020-07-28 Workfusion, Inc. Task similarity clusters for worker assessment
CN110853557B (en) * 2019-11-26 2022-09-02 武汉帆茂电子科技有限公司 FPGA-based liquid crystal module bomb inspection system and inspection method
CN116016293B (en) * 2023-03-22 2023-08-25 深圳市亿联无限科技股份有限公司 Multi-task queue testing method, system and platform for network communication product production

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030050829A1 (en) * 2001-09-13 2003-03-13 International Business Machines Corporation Method and system for collecting and distributing data evaluating the job performances of short term job contractors through a computer controlled centralized database
US20050154600A1 (en) * 2004-01-08 2005-07-14 American International Group, Inc. Extended work program
US20070124363A1 (en) * 2004-07-21 2007-05-31 The Mathworks, Inc. Instrument-based distributed computing systems
US7778865B1 (en) * 2003-05-30 2010-08-17 Kane Jeffrey S Distributional assessment system
US20110313820A1 (en) * 2010-06-17 2011-12-22 CrowdFlower, Inc. Using virtual currency to compensate workers in a crowdsourced task

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030050829A1 (en) * 2001-09-13 2003-03-13 International Business Machines Corporation Method and system for collecting and distributing data evaluating the job performances of short term job contractors through a computer controlled centralized database
US7778865B1 (en) * 2003-05-30 2010-08-17 Kane Jeffrey S Distributional assessment system
US20050154600A1 (en) * 2004-01-08 2005-07-14 American International Group, Inc. Extended work program
US20070124363A1 (en) * 2004-07-21 2007-05-31 The Mathworks, Inc. Instrument-based distributed computing systems
US20110313820A1 (en) * 2010-06-17 2011-12-22 CrowdFlower, Inc. Using virtual currency to compensate workers in a crowdsourced task

Also Published As

Publication number Publication date
WO2013003861A2 (en) 2013-01-03

Similar Documents

Publication Publication Date Title
WO2011159433A3 (en) Using virtual currency to compensate workers in a crowdsourced task
EP3385889A4 (en) Abnormality detection system, abnormality detection method, abnormality detection program, and method for generating learned model
EP3096235A4 (en) Information processing system, information processing server, information processing program, and fatigue evaluation method
EP2657863A3 (en) Methods and computer-program products for generating grasp patterns for use by a robot
MY186462A (en) Training ensurance method and system for computer directed assembly and manufacturing
AU2012272977A8 (en) System and method for building and managing user experience for computer software interfaces
DE102011105141A8 (en) METHOD AND SYSTEM FOR SIMULATING A WORKING PROCESS ON A TOOL MACHINE
WO2011159434A3 (en) Distributing a task to multiple workers over a network for completion while providing quality control
DE112012003977T8 (en) Non-intrusive method and apparatus for automatically distributing security rules in a cloud environment
MX2013001171A (en) System and method for determining a lubricant discard interval.
DE102014103531A8 (en) METHOD AND DEVICE FOR REGULATING A WORKFLOW IN A PROCESS ENGINEERING SYSTEM
WO2015101461A3 (en) Method and apparatus for design of a metrology target
WO2012131329A3 (en) Improvements in and relating to methods and systems for investigating radioactive sources in locations
WO2013188511A3 (en) System and method for online hiring and virtual interview questioning
WO2014184710A3 (en) Virtual reality based welding simulator
EP3327701A4 (en) Online test method and online test server for evaluating creativity for ideas
MX2016003852A (en) Platform for generating personalized career pathways.
GB201219722D0 (en) X-ray analysis apparatus
EP3327717A4 (en) Information processing device, method for evaluating fatigue level, and program
GB201309785D0 (en) Methods and systems for providing real-time information regarding objects in a network
WO2013003861A3 (en) Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks
WO2014052786A3 (en) Continuous three-dimensional human-work planning simulation
GB201206932D0 (en) Comparing event data sets
MY193347A (en) Mantle phenotype detection in palm
WO2014077993A3 (en) Distributed spreading tools and methods

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12804489

Country of ref document: EP

Kind code of ref document: A2