How to Manage HITs

The following suggestions are based on my experiences, and the feedback received from thousands of Workers.

Make sure you pilot test your HITs.  Complete it yourself using your Worker account.  

• Proofread, proofread, proofread.
• Check links and videos, and make sure a completion code is being provided (if you are using one). Check to make sure the submit button is available.  
• Test the HIT in multiple browsers.  
• Allow at least one Worker to complete the HIT and provide feedback.

Approve submissions as soon as possible, preferably within 5 days.  

• Utilize the option to reward bonuses, especially if you are trying to form a group of Workers with a specific skill.  Bonuses are always appreciated!

Communicate with Workers.

• Send e-mails to recruit Workers who qualify to take your other HITs.
• Respond to e-mails as soon as possible.  Workers can't move forward until you answer their questions.  
• High-volume Requesters should provide a quicker way to communicate with Workers (e.g., instant messaging).

Listen to Workers' suggestions, feedback, and criticisms.  Monitor your reputation by visiting popular MTurk forums.

Avoid rejecting work because it was completed "too quickly" or because the Worker failed the attention check.  This is especially important when you qualified them to take the HIT that they are now being rejected for completing.  

• E-mail Workers before rejecting their work to avoid misunderstandings and misinterpretations.  There may be a problem with your HIT, and you won't know unless you ask Workers.
• If you must reject a submission, then provide detailed feedback about why it is being rejected. 

Share the results of your study with Workers who are interested.  

How to Design HIT Layouts

The following suggestions are based on my experiences, and the feedback received from thousands of Workers.

When possible, design the task within the HIT window.  

• If this is not possible, then make sure you choose external websites that supported by multiple operating systems and browsers (such as Qualtrics or Survey Monkey).  

• Add target="_blank" to the HTML code for these external links so they open a new window to minimize disruption to the Worker.

Make HITs fun and interesting.    

• Use games, puzzles, and stories to get the information you want.  

• Include images and visuals whenever possible (e.g., on the "Thank You" page, periodically throughout a survey).  

Be clear with your instructions.

• Show all of the steps required to complete the HIT.  Use numbered lists and visual aids.  Ask someone else to read them in the pilot test.  For more complicated HITs, include a short video to demonstrate how to do the HIT.

• Specify how much time it takes to complete the HIT, based on the average time from your pilot test (see below).

• Be clear about whether Workers can complete more than one HIT in a batch.  

• Explain under what circumstances a HIT will be rejected.  Tell Workers the time frame for approving/rejecting submissions (e.g., 24 hours, 3 days, 1 week).

• Tell Workers who to contact if they have problems.

Design HITs to be "user friendly".

• The format should be simple.  Use font that is size 12 or larger, and avoid unusual color combinations (e.g., yellow font on a black background).

• Allow Workers to check their eligibility before accepting a HIT.  Use a Worker ID Check or provide a list of ineligible Worker IDs.

• Include a progress bar so Workers know how to pace themselves.

• Limit scrolling and limit the total number of pages. Avoid putting only one item on each page, and avoid putting all of your items on one page. Balance the use of "bubbles" (multiple choice questions) and open-ended questions.

• When asking about potentially sensitive information (e.g., gender, race, religion, etc.), provide response choices like "Other" or "Prefer not to answer".

• Be careful with attention checks.  Avoid quizzing Workers as a way of checking whether they read something (unless you told them to remember it).

• Specify whether a task will require Workers to listen to audio and video clips. Some individuals are deaf, and could benefit from this information up front.

If you are using an external website (e.g., Survey Monkey) to collect information from Workers, the last page of the survey should include a debriefing page to explain what the HIT was about, contact information, and completion codes.

How to Design HIT Properties

The following suggestions are based on my experiences, and the feedback received from thousands of Workers.

Create unique titles for your HITs, and clearly describe what you are asking Workers to do.  

• Use phrases like "Invitation Only" and "Bonus Offer" to communicate and attract the appropriate people. 
• When you want to launch a batch of HITs, make sure the following information is exactly the same for the HITs you want to include in the batch:  title; description; keywords; reward per assignment; results are automatically approved in; time allotted per assignment; and masters' and additional qualifications.

Pay at least 10 cents per minute.  Workers are more likely to be interested in the HIT, which means they are more likely to be thorough and conscientious when completing it. It also helps you build a positive reputation, which can benefit the quality and speed of data collection.

Give Workers plenty of time to complete your HITs.  Several Workers recommended at least 24 hours.  They can accept the HIT, and come back to it later when they are ready.

Be careful with assigning qualifications.  

• They are all or nothing, meaning only Workers who have all of them can participate.  For example, assign only 1 'Location' qualification.  If you set 2 locations, then no Worker qualifies (because no worker is from 2 locations). 

• Use Workers' approval totals and ratings as qualifiers to weed out "scammers", but don't set the requirements so high that you eliminate most of the workforce.