A new article on 90 Percent of Everything discusses a few ways to screen out potential "fake users" who lie about their qualifications to participate in your study:

In fact, a lot of liars can be screened out by writing a really good screener questionnaire. For example, here's a decoy question that the Mozilla metrics team used in their recent Test Pilot survey.

This is really handy to know when live recruiting users from your website--the risk for fakers is even higher when the recruiting pool consists of anyone who comes to your website. Chapter 3 of our Remote Research book also touches on this subject, specifically as it relates to live recruiting. Here are two more pointers:

--Occasionally when people catch wind of a paid survey offer, they like to post it on "bargain hunting sites" like FatWallet. If you get a sudden surge of recruits, that may be the reason; check the referrer data in your traffic log or analytics to confirm where users are coming from.

--Use open-ended questions to test people's motives for coming to the site. If someone responds to the question "Why did you come to the site today?" with a vague answer like "To check the offerings" or "Just looking around", consider that a yellow flag, and follow up with more specific interview questions.