Identify Malicious Labelers among Crowdsourcing Participants
Evaluating Crowd-sourcing Participants in the Absence of Ground-Truth
The code presents the novel approach to identify unhelpful or adversarial participants in a crowdsourcing setting where multiple labelers are available. Further, the code demonstrates the robustness of the perfected scoring scheme to evaluate the participants.
Note:
(a) Constant features beta and gamma are the last dimension of alpha and W respectively.
(b) Resulting probabilities [0, 1] from the model shall be mapped to [-1 1] using the simple transformation, 2 x y{0,1} - 1 = y{-1,1}.
Folders ErrorStatistics, MultiLabelerMethods and the DATA were shared by Yan Yan, now a Senior Software Engineer at LinkedIn and Romer Rosales, Principal Scientist at LinkedIn.
This code is licensed under the LGPLv3 license. Please feel free to use the code in your research and development works. We would appreciate a citation to the paper below when this code is helpful in obtaining results in your future publications.
Publication for citation:
(1) R. Subramanian, Romer Rosales, Glenn Fung, Prof. Jennifer Dy, "Evaluating Crowdsourcing Participants in the Absence of Ground-Truth", at the 26th Annual Conference on Neural Information Processing Systems (NIPS 2012) Workshop on Human Computation for Science and Computational Sustainability, Accepted Date: Oct 7, 2012.
(2) Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy, "Learning from Multiple Annotators with Varying Expertise", Machine Learning Journal, June 2014, Volume 95, Issue 3, pp 291-327.
Citar como
Ram Subramanian (2024). Identify Malicious Labelers among Crowdsourcing Participants (https://github.com/linux-ram/Evaluate-Crowd-Participant), GitHub. Recuperado .
Compatibilidad con la versión de MATLAB
Compatibilidad con las plataformas
Windows macOS LinuxEtiquetas
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!Descubra Live Editor
Cree scripts con código, salida y texto formateado en un documento ejecutable.
ErrorStatistics
MultiLabelerMethods
No se pueden descargar versiones que utilicen la rama predeterminada de GitHub
Versión | Publicado | Notas de la versión | |
---|---|---|---|
1.0.3 | Added project image |
|
|
1.0.2 | Title Change |
|
|
1.0.1 | Title Change |
|
|
1.0.0 |
|