qiang_liu
The title is inference over the collective labeling processing.
 A factor graph was built to model the accuracy of the workers and tasks.
 Assume most workers are correct

Using different priors can be interpretated
 Deterministic prior > Majority voting
 Haldane prior > KOS

Bluebird dataset
 Majority voting
 KOS
 Welinder et al, 2010
 Proposed (simple but good results)

CrowdScale Challenge at HCOMP 2013
 Google Data, CrowdFlower Data
 Unbalanced labels
 Ambiguous labels

Web search relevance (MSR)
 Ordinal rating (ICML 2014)
Combing Experts and Crowds
Not assuming Majority of the crowds are correct.
Using control questions to deal with bias by crowds.
Two words captral. One is known, the other is for collection data.
Two stage estimator
 Scoring the workers based on the control question.
 Predicting target items.
Joint Estimator
 Jointly modeling [Z, V] = argmax p(Z, T  z*, L)
Question:
 theoretical properties
 How to allocate control questions. (NIPS 2013)
Workers are asked l questions, among which k are control questions. > What's the best k, under buget constraints.
?? 36, why increasing k will increase MSE at some point? the MSE is the overall error.
Theoretically (NIPS 2013) by asytonical analysis.
when bias model is Gaussian bias model,
 Twostage estimator > k* = sqrt(l)

Joint estimator > k* = l / sqrt(n) n: total number of target questions. n is involved because in joint inference, the information in the noisy labeling can be leveraged.
 Assume regular random bipartite graph > using the spectual property of the bipartite to derive
Dataset,
 Point Spreads of NFL Games
Another question:
 How to choose tasks to label for Experts.
> combinatoral optimization
Application:
 MOOCS, crowding grading and teachers check some of them to propergate belief over the crowds.
A good Question: > Whether feedback to crowds will improve the results. You are doing good, bad, poor.