Survey Design
This article is intended primarily for those who are new to survey research. It discusses options and provides suggestions on how to design and conduct a successful survey project.
It does not provide instruction on using specific parts of The Survey System, although it mentions parts of the program that can help you with certain tasks.
The Steps in a Survey Projec:
1. Establish the goals of the project - What you want to learn
2. Determine your sample - Whom you will interview
3. Choose interviewing methodology - How you will interview
4. Create your questionnaire - What you will ask
5. Pre-test the questionnaire, if practical - Test the questions
6. Conduct interviews and enter data - Ask the questions
7. Analyze the data - Produce the reports
Establishing Goal:
The first step in any survey is deciding what you want to learn. The goals of the project determine whom you will survey and what you will ask them.
If your goals are unclear, the results will probably be unclear. Some typical goals include learning more about:
• The potential market for a new product or service
• Ratings of current products or services
• Employee attitudes
• Customer/patient satisfaction levels
• Reader/viewer/listener opinions
• Association member opinions
• Opinions about political candidates or issues
• Corporate images
These sample goals represent general areas. The more specific you can make your goals, the easier it will be to get usable answers.
Selecting Your Sample:
There are two main components in determining whom you will interview.
The first is deciding what kind of people to interview. Researchers often call this group the target population. If you conduct an employee attitude survey or an association membership survey, the population is obvious.
If you are trying to determine the likely success of a product, the target population may be less obvious. Correctly determining the target population is critical.
If you do not interview the right kinds of people, you will not successfully meet your goals.
The next thing to decide is how many people you need to interview.
Statisticians know that a small, representative sample will reflect the group from which it is drawn.
The larger the sample, the more precisely it reflects the target group.
However, the rate of improvement in the precision decreases as your sample size increases. For example, to increase a sample from 250 to 1,000 only doubles the precision. You must make a decision about your sample size based on factors such as:
time available, budget and necessary degree of precision.
Avoiding a Biased Sample:
A biased sample will produce biased results. Totally excluding all bias is almost impossible; however, if you recognize bias exists you can intuitively discount some of the answers.
The following list shows some examples of biased samples.
The consequences of a source of bias depend on the nature of the survey. For example, a survey for a product aimed at retirees will not be as biased by daytime interviews as will a general public opinion survey. A survey about Internet products can safely ignore people who do not use the Internet.
Quotas:
A Quota is a sample size for a sub-group. It is sometimes useful to establish quotas to ensure that your sample accurately reflects relevant sub-groups in your target population.
For example, men and women have somewhat different opinions in many areas. If you want your survey to accurately reflect the general population's opinions, you will want to ensure that the percentage of men and women in your sample reflect their percentages of the general population.
If you are interviewing users of a particular type of product, you probably want to ensure that users of the different current brands are represented in proportions that approximate the current market share. Alternatively, you may want to ensure that you have enough users of each brand to be able to analyze the users of each brand as a separate group.
If you are doing telephone or Web page interviewing, The Survey System's optional Sample Management or Internet Module can help you enforce quotas. They let you create automatically enforced quotas and/or monitor your sample during interviewing sessions.
Interviewing Methods
Once you have decided on your sample you must decide on your method of data collection.
Each method has advantages and disadvantages.
Personal Interviews
An interview is called personal when the Interviewer asks the questions face-to-face with the Interviewee. Personal interviews can take place in the home, at a shopping mall, on the street, outside a movie theater or polling place, and so on.
Advantages
• The ability to let the Interviewee see, feel and/or taste a product.
• The ability to find the target population. For example, you can find people who have seen a film much more easily outside a theater in which it is playing than by calling phone numbers at random.
• Longer interviews are sometimes tolerated. Particularly with in-home interviews that have been arranged in advance. People may be willing to talk longer face-to-face than to someone on the phone.
Disadvantages
• Personal interviews usually cost more per interview than other methods. This is particularly true of in-home interviews, where travel time is a major factor.
• Each mall has its own characteristics. It draws its clientele from a specific geographic area surrounding it, and its shop profile also influences the type of client. These characteristics may differ from the target population and create a non-representative sample.
Telephone Surveys
Surveying by telephone is the most popular interviewing method in the USA. This is made possible by nearly universal coverage (96% of homes have a telephone).
Advantages
• People can usually be contacted faster over the telephone than with other methods. If the Interviewers are using CATI (computer-assisted telephone interviewing), the results can be available minutes after completing the last interview.
• You can dial random telephone numbers when you do not have the actual telephone numbers of potential respondents.
• CATI software, such as The Survey System, makes complex questionnaires practical by offering many logic options. It can automatically skip questions, perform calculations and modify questions based on the answers to earlier questions. It can check the logical consistency of answers and can present questions or answers choices in a random order (the last two are sometimes important for reasons described later).
• Skilled interviewers can often elicit longer or more complete answers than people will give on their own to mail, email surveys (though some people will give longer answers to Web page surveys). Interviewers can also ask for clarification of unclear responses.
• Some software, such as The Survey System, can combine survey answers with pre-existing information you have about the people being interviewed.
Disadvantages
• Many telemarketers have given legitimate research a bad name by claiming to be doing research when they start a sales call. Consequently, many people are reluctant to answer phone interviews and use their answering machines to screen calls. Since over half of the homes in the USA have answering machines, this problem is getting worse.
• The growing number of working women often means that no one is home during the day. This limits calling time to a "window" of about 6-9 p.m. (when you can be sure to interrupt dinner or a favorite TV program).
• You cannot show or sample products by phone.
Mail Survey:
Advantages
• Mail surveys are among the least expensive.
• This is the only kind of survey you can do if you have the names and addresses of the target population, but not their telephone numbers.
• The questionnaire can include pictures - something that is not possible over the phone.
• Mail surveys allow the respondent to answer at their leisure, rather than at the often inconvenient moment they are contacted for a phone or personal interview. For this reason, they are not considered as intrusive as other kinds of interviews.
Disadvantages
• Time! Mail surveys take longer than other kinds. You will need to wait several weeks after mailing out questionnaires before you can be sure that you have gotten most of the responses.
• In populations of lower educational and literacy levels, response rates to mail surveys are often too small to be useful.
This, in effect, eliminates many immigrant populations that form substantial markets in many areas. Even in well-educated populations, response rates vary from as low as 3% up to 90%. As a rule of thumb, the best response levels are achieved from highly-educated people and people with a particular interest in the subject (which, depending on your target population, could lead to a biased sample).
One way of improving response rates to mail surveys is to mail a postcard telling your sample to watch for a questionnaire in the next week or two.
Another is to follow up a questionnaire mailing after a couple of weeks with a card asking people to return the questionnaire. The downside is that this doubles or triples your mailing cost.
If you have purchased a mailing list from a supplier, you may also have to pay a second (and third) use fee - you often cannot buy the list once and re-use it.
Another way to increase responses to mail surveys is to use an incentive. One possibility is to send a dollar bill (or more) along with the survey (or offer to donate the dollar to a charity specified by the respondent).
If you do so, be sure to say that the dollar is a way of saying "thanks," rather than payment for their time.
Many people will consider their time worth more than a dollar. Another possibility is to include the people who return completed surveys in a drawing for a prize.
A third is to offer a copy of the (non-confidential) result highlights to those who complete the questionnaire. Any of these techniques will increase the response rates.
Remember that if you want a sample of 1,000 people, and you estimate a 10% response level, you need to mail 10,000 questionnaires.
You may want to check with your local post office about bulk mail rates - you can save on postage using this mailing method. However, most researchers do not use bulk mail, because many people associate "bulk" with "junk" and will throw it out without opening the envelope, lowering your response rate.
Also bulk mail moves slowly, increasing the time needed to complete your project.
11:48 AM
Unknown
Posted in: 
