The title is inference over the collective labeling processing.

Combing Experts and Crowds

Not assuming Majority of the crowds are correct.

Using control questions to deal with bias by crowds.

Two words captral. One is known, the other is for collection data.

Two stage estimator

Joint Estimator


Workers are asked l questions, among which k are control questions. --> What's the best k, under buget constraints.

?? 36, why increasing k will increase MSE at some point? the MSE is the overall error.

Theoretically (NIPS 2013) by asytonical analysis.

when bias model is Gaussian bias model,


Another question:

  1. How to choose tasks to label for Experts.

---> combinatoral optimization


A good Question: ---> Whether feedback to crowds will improve the results. You are doing good, bad, poor.