image thumbnail

Identify Malicious Labelers among Crowdsourcing Participants

version 1.0.3 (326 KB) by Ram Subramanian
Evaluating Crowdsourcing Participants in the Absence of Ground-Truth

2 Downloads

Updated 12 Jun 2018

From GitHub

View license on GitHub

Evaluating Crowd-sourcing Participants in the Absence of Ground-Truth

The code presents the novel approach to identify unhelpful or adversarial participants in a crowdsourcing setting where multiple labelers are available. Further, the code demonstrates the robustness of the perfected scoring scheme to evaluate the participants.

Note:
(a) Constant features beta and gamma are the last dimension of alpha and W respectively.
(b) Resulting probabilities [0, 1] from the model shall be mapped to [-1 1] using the simple transformation, 2 x y{0,1} - 1 = y{-1,1}.

Folders ErrorStatistics, MultiLabelerMethods and the DATA were shared by Yan Yan, now a Senior Software Engineer at LinkedIn and Romer Rosales, Principal Scientist at LinkedIn.

This code is licensed under the LGPLv3 license. Please feel free to use the code in your research and development works. We would appreciate a citation to the paper below when this code is helpful in obtaining results in your future publications.

Publication for citation:
(1) R. Subramanian, Romer Rosales, Glenn Fung, Prof. Jennifer Dy, "Evaluating Crowdsourcing Participants in the Absence of Ground-Truth", at the 26th Annual Conference on Neural Information Processing Systems (NIPS 2012) Workshop on Human Computation for Science and Computational Sustainability, Accepted Date: Oct 7, 2012.

(2) Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy, "Learning from Multiple Annotators with Varying Expertise", Machine Learning Journal, June 2014, Volume 95, Issue 3, pp 291-327.

Cite As

Ram Subramanian (2021). Identify Malicious Labelers among Crowdsourcing Participants (https://github.com/linux-ram/Evaluate-Crowd-Participant), GitHub. Retrieved .

MATLAB Release Compatibility
Created with R2017a
Compatible with any release
Platform Compatibility
Windows macOS Linux

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!
To view or report issues in this GitHub add-on, visit the GitHub Repository.
To view or report issues in this GitHub add-on, visit the GitHub Repository.