How to Collect High Quality Data

Use a screening survey to identify a recruitment pool of Workers who meet your criteria for participating in your research.  This could be especially useful if you plan to use Mechanical Turk to collect data for multiple research projects.  Click here for more information about screening surveys.

• Use open-ended items and lists of options for close-ended items to reduce the chances of someone correctly guessing your criteria.  For example, the question "Are you currently employed at least 40 hours per week?" may lead some people to guess your criteria and then respond based on their assumptions about who you want to recruit.  The question, "How many hours do you currently work at your current employer?" forces the participant to count their hours, allows the researcher to choose who qualifies with more confidence, and provides more variation in the data collected (a bonus for statistical analysis).

Include quality control items to reduce the chances people are mindlessly clicking on response options.  An example of a quality control item is, “Please select ‘strongly agree’ if a human is completing this survey.”  To reduce people's often negative reaction to items like these, inform them upfront that these items will appear, and that they are encouraged to be fully engaged when they participate in your research.  

Conduct a pilot study to assess the participation experience, and to make improvements early on in the process.  

• Include a text box in each HIT that allows Workers to provide their feedback regarding such things as confusing items or instructions that may comprise the data.  Click here for the HTML code.