Personalization of software could help improve users’ productivity and efficiency and enable higher rate of personalized software adoptability among users. Machine Learning could be effective in enabling personalization. However, personalization achieved through Machine Learning demands lots of user-specific training labels. Latest forms of Machine Learning (e.g., Online Active learning algorithms) try to reduce the number of labels that users need to provide for efficient personalization. However, existing online active learning techniques consider humans as oracles and assume that humans are a constant source of quality labels through time. Common users typically do not provide high quality labels consistently, and it is important to study human behavior when they are part of the loop to improve the quality of labels.