The questionnaire is the most commonly used method of gathering information about use and users. Questionnaires are easy to prepare and it is surprisingly easy to prepare a bad one. It is important to be able to avoid major pitfalls and to be able to identify them in the research done by others.
There are four major methods of information-gathering:
Each of these methods has distinct assets and liabilities. The thoughtful research will be familiar with these or with resources that provide sound advice. It is important to relate the data gathering method to the nature of the population or sample, both size and dispersion. Interviewing is much more expensive and requires skilled interviewers.
Questionnaires are a less expensive way to reach more people, including people at some distance. Depending upon the mode of distribution, this can be quickly done and data analysis can begin right away. The questionnaire avoids interviewer bias, guiding, and cues that can impact the validity and reliability of the data collection. Anonymity insures more valid responses. Response quality is better because respondents may gather and consult sources needed to respond well.
The use of questionnaires for a variety of commercial purposes as well as for "real" research have created a situation in which response rates have fallen over time. A response rate between 25 and 33 percent is now considered to be a good one. Even after follow-ups, the response rate may still be low. It does require some motivation to respond and then the respondent must understand the questions and the sort of response required. Because the researcher is at some distance, there is little opportunity to develop rapport with the respondent. There is no opportunity to probe or clarify. Misunderstandings about purpose, questions, privacy, or whatever cannot be answered.
While the process is simple and straight-forward, there are many opportunities for error.
We are asking about a person's knowledge, attitudes, beliefs, feelings, motivations, anticipations, future plans or past behavior. What can we do to motivate people to respond and to respond truthfully? For example, one older study found that about 20 percent of those without library cards claimed to have one when asked. There is a strong tendency to give answers that are socially desirable, make the respondent look good, or that will please the researcher. While we begin with an assumption of truthfulness, it may be useful to use variants of the same question to capture more of reality.
Another concern is whether or not the respondent knows enough to provide a meaningful answer. Some one who has never used the teen collection may find it difficult to indicate if it is valued.
Validity is the degree to which we are measuring what we need to measure. The questionnaire should gather valid responses. Reliability is the degree to which we receive the same measurement over time. Would we receive the same response if the respondent had answered the questionnaire earlier or later?
At the beginning and throughout the process, you need to consider these questions:
If possible, begin with another's questionnaire, especially if you are doing a use and user survey [do receive permission first]. This is less expensive, but there are other advantages. Others should have validated their instrument. You will be able to compare your findings with theirs and built upon previous generalizations. Knowledge can cumulate. The ERIC data base includes a variety of questionnaires used in previous use and user surveys. Library Literature is also quite useful via the --survey tag. Ideally, you would keep changes to a minimum to facilitate comparison, but you may build upon an existing instrument by adding additional questions or making essential changes.
Ordinarily, there are six steps:
A few deep thoughts are needed at the beginning:
The number of questions must be limited to insure a good response. Response rate declines rapidly as the number of questions, especially those that require time and thought are added. Questions can usually be divided into two categories: (1) absolutely necessary and (2) interesting. Be certain that the information to be gathered is not available elsewhere, i.e. in census data or another report. Respondents are much less likely to respond to a question if they feel the answer is readily available.
Questions will need to be placed in a logical sequence.
Typically, information sought falls into four categories:
Attitude questions ask people to indicate if they favor or oppose, if they prefer or not, should or should not, right versus wrong, desirable versus undesirable. These questions require sensitive, thoughtful wording.
Belief questions ask people if something is true or false, correct or incorrect, accurate or inaccurate.
Behavior questions ask people what they have done, what they do, or what they plan to do.
Attribute or demographic questions ask about age, income, education, and the like.
Open-ended questions provide no answer choices. They are easy to ask and allow for a wide variety of responses, including the creative and unusual. Open-ended questions are especially useful when you don't know the likely values or can't anticipate how the respondent will respond. The information gathered by open-ended questions could then be used to develop appropriate close-ended questions for another questionnaire.
These questions force the respondent to think and allow the respondent to clarify and explain a response. If the respondent takes needed time and makes the effort, responses can be illuminating and yield much useful information.
The response rate will be lower because the blank space is demanding and intimidating, especially for those who don't like to write. Illegible handwriting may be a problem.
Since responses are not really ordered, analysis requires considerable time and effort. It may be difficult to measure and classify responses. Responses may be off base because there is inadequate guidance from the instrument itself.
These questions provide specific answer choices although there may be an "other" value with brief space for adding an additional value. With close-ended questions, there is always the possibility that the right question will not be asked and valuable information will not be gathered. GIGO certainly applies here.
Ordered close-ended questions require respondents to select a particular response. The responses are easily selected. They take little time, at least by most respondents.
These questions require well defined variables and values. They work best when there are a small number of reasonable answer possibilities.
Unordered questions ask respondents to rank values and are useful for identifying priorities. Requires well defined variables and values. Unordered close-ended questions are not reliable if there are more than five values. Most respondents find ranking after one or two values to be difficult.
Criteria for ranking must be clearly identified and the order of the values must make sense to the respondent.
These questions may be ordered or not, but they do include the "other" option which adds some flexibility and provides the opportunity to add information not otherwise captured by the instrument.
More flexibility may mean better and more valid responses. New values may be selected from the leading "other" values, but this will make analysis more challenging
Close-ended questions may be categorized by the number of values. Two value questions are dichotomous. These questions are easy to ask and are quickly answered. Analysis is straight-forward and quick.
However, two alternatives are usually not enough. Some times, respondents will select the first value so the ordering of the values has some impact. Each value must be exclusive. The researcher must know the notable alternatives.
Multiple choice questions [more than two values]are also easy to analyze, but do require more time and effort. These questions require more time and effort by the respondent. Typically, respondents tend to choose from the top or middle of the list.
The first step is to visualize the audience. Next ask two questions:
Avoid wrong words. Wrong words are vague, too specific, misunderstood, objectionable, irrelevant, and uninteresting. It takes time to select the right word. One researcher went through 41 versions of a question before the words were just right.
Insure that words are uniformly understood and provide brief definitions if needed. For example, how will the patron define "use" when asked questions about the degree of library or information center use. Words like "often," "rarely," "few," and "many" are used flexibly and different people have very different ideas as to what these words mean.
Use simpler, but specific words. For example, "free-time" is better than "leisure time." Substitute specifics for "several," "most," "usually," and the like to insure better reliability. When possible, use shorter words. Clearly define professional words if the audience is not professional, for example "volume" and "bibliographic instruction" may mean little to most college students. Insure that each word in a question is necessary. What would happen if that word was removed.
Avoid abbreviations and initialisms. If used, do spell out in the first mention, and again later if at some distance from the first mention. Read and reread for directness and clarity. For example, how clear is "What changes should the government make in its policies toward libraries?"
Avoid the too specific and use ranges instead. Few respondents will know how many books they have checked out from a public library in the last year.
Avoid value-oriented words that might bias the response, i.e. "should the library collection contain filthy books?"
Avoid self-incriminating questions such as "have you ever abused [definition needed] library materials" and attempt a more subtle approach. For example,
Avoid double questions such as "are reference librarians friendly and knowledgeable" and use two separate questions instead.
Avoid negatives such as "should the librarian not be directly responsible to the city manager?"
Insure that choices are exclusive. For example, there is a problem if age choices are " 18-35 years" and "35-50" years. Similarly, asking "how did you learn about our new library service" with the values "from a friend or relative," "at work," or "from the newspaper" is problematic because the values are not exclusive.
Avoid questions that assume too much knowledge such as "do you agree with the governor's stand on regional libraries?"
Do provide appropriate time referents such as "how many times have you gone to the library in November of this year?"
Avoid speculative questions because the response are often not reliable. For example, the question "if the library provided a full-text database on blumpf, how often would you use it?" is likely to yield information of minimal value.
Revise the wording of each question until it is just right.
Do include an option for "don't know. Double check to insure that values are exclusive and independent. Balance scales used in close-ended questions with an equal number on each side of the middle position. It is better to arrange values vertically rather than horizontally to eliminate the between the value response.
There are at least three different approaches for you to consider. One is to begin with easy questions in order to build confidence and make the respondent comfortable. The second is to place the more important questions first to motivate and give a sense that the questionnaire is important and well worth the respondent's time. The third is to place general questions first with the more specific ones to follow.
There is some disagreement over the placement of demographic questions. One approach is place them at the beginning because they set the stage and are easily answered. The other approach is to place them at the end because some respondents don't like to answer any "personal" questions. An incomplete response would still yield some useful data.
It is usually helpful to group questions by the type of response required. This makes responding easier. There should be a logical transition, with appropriate text, between question groups.
Objectionable, time-consuming, or especially difficult questions should be at the end. These questions are less likely to skew responses to preceding questions. Respondents have an investment in responding to the questionnaire and are less likely to quit.
Selecting the first question is crucial. It should be clearly related to the problem, be interesting to the respondent, and be easy to respond to. The question should be objective or neutral. It should apply to everyone in the population . Finally, it should establish a visual or graphic precedent for the questions that follow.
Size, shape, weight, color, paper quality, design, and layout all answer these important questions:
The first impression is crucial in improving the response rate.
Your job is to make it as easy as possible to complete the questionnaire quickly and with minimal effort. Consider the traditional negatives associated with Questionnaires:
Avoid gimmicks or unusually creative formats.
Use white or off-white paper with black or blue ink. The paper needs to be light enough to reduce postage, but not so light as to appear cheap.
There should be no errors.
Printing or reproduction should be of letter quality and appear original. Print should be dark and clear.
Response spaces should be in the same position on each question and pushed toward the left or right margin where they are easily found. Place column numbers for coding near the question number and in parenthesis. Consider data analysis from the beginning. Construct your code book as soon as your questionnaire has finished pre-testing. All questions and pages should be clearly numbered. If both sides of the paper are used, a large, bold, OVER must appear at the right footer of each verso page.
Your return address should appear at the foot of the questionnaire and probably at the head as well. The study title should be clear, understandable to lay people and in a larger, bold font.
Clear, brief instructions are needed at the head and in the body as needed. Deadline information may or may not be useful. It is an asset by encouraging respondents to reply promptly. It is a liability of potential respondents decide not to reply because the deadline is near or past.
Do provide a reasonable space at the end for comments. Also at the end, indicate what will happen next with the study. If dealing with a professional audience, indicate that a copy of the study will be available and how they might get a copy.
This is a slow process and must be done with care. Pages should be neat and aesthetically pleasing.
Use lower case letters for the questions and upper case letters for the answers [responses]. Each value should be numbered and the number should be circled. Use the same number throughout for the same value. For example, yes should always be "1." Do not split questions between pages.
Establish vertical flow with the beginning of the question and the beginning of the answers. Leave enough white space between questions. Directions should appear in parentheses so that they stand out. You may use multiple columns and hats to conserve space as in the example below.
FATHER................................. MOTHER
1............................................... 1................. NO FORMAL EDUCATION
2............................................... 2 .................SOME GRADE SCHOOL.
Skipping not applicable questions is always a problem. Use arrows to direct to next applicable question. Indent questions dependent on an earlier response. Use text boxes to direct respondents to the next applicable question.
Use transitions for continuity and to stimulate motivation. Transitions are needed for new topics and to break monotony.
The opening is extremely important. It should be short, interesting, and stimulate response. In particular, you must convincingly demonstrate why the study is important, why their response is especially important, and how they will benefit from the study. The title of the study, clearly understandable by the lay person, must be included. The name of the principal investigator and the institutional address should appear here.
The close is also important. There should be an opportunity for comment as well as a thank you for participation. The name of the PI and contact information should also be included.
Shorter is better. Ask no unnecessary questions. Ask only questions that can be answered. If the question is longer, it must look easy to respond to. If the audience is really interested in the topic, length is less important.
Pre-testing is absolutely necessary. Don't be too eager to begin collecting data. Here are typical questions that pre-testing should answer:
Pre testing may be done by three types of people. Colleagues are a logical choice, but may give you only a superficial examination. Too, that may not want to offend you with critical comment. Potential users of the data should have substantive knowledge of the topic. Finally, and best, are people drawn from the population to be surveyed. These people cannot then be used in the final study.
Revise and continue to test until your are satisfied that the questionnaire is as good as it can be.
These are not typical, but they may be useful in stimulating a better response rate. The evidence regarding cost - benefit is mixed. This alert letter -- the questionnaire is coming -- should include:
The major liability of the questionnaire is the low response rate. Causes of poor response include:
There are three typical appeal elements. The first is essentially an appeal for help or "will you do me a favor?" Here, we emphasize our personal appreciation and gratitude. This is a weak appeal if much time and effort is required. Follow-ups can be difficult.
The second appeal focuses on a shared problem that we must solve together. Their help is needed to find a solution that will improve some aspect of their life. Here we have an exchange relationship with each helping the other.
The third appeal element is rarely used by information professional researchers. Here an incentive is included to help compensate the respondent for her time. These incentives are usually inexpensive tokens, but winning a prize may make a difference. Dr. Bill once won a $100.00 Amazon gift certificate for participating in a study.
You will need to decide how formal or informal you will be in the cover letter. You want to personalize the letter and establish rapport, but you don't want to be too familiar, especially with people you have not met.
An endorsement by an individual or group well known to your audience can increase response rates and is well worth the effort. Can you identify some endorsement possibilities?
The letter should look like an original with letterhead and an original-looking signature.
The cover letter should be limited to a single page.
The first paragraph should explain what the study is about and convince the respondent that the study is important and useful. How does the study relate to their problems? Why will the study make a difference? How will the results be used? It is best to avoid the words "questionnaire" and "survey" in this paragraph because of their negative associations.
The second paragraph focuses on why the particular respondent is important to the success of the study. Why were you selected? Why the population segment that you represent is important? Who should complete the instrument?
The third paragraph considers the questionnaire itself. It will not take too long to complete and the questions themselves are not difficult. We also emphasize that all responses are confidential and anonymous.
The fourth paragraph looks at social utility. We focus on the difference that the responses may make and how the results will be used to affect change. We would also promise respondents summary copies of the results, perhaps via a particular URL.
The fifth paragraph closes with a statement that the investigator is willing to answer any questions or concerns. There is also a thank you statement.
The envelope must appear professional and be clearly distinguished from junk mail. It should the correct size to hold the questionnaire without unnecessary folding.
First class mail is best because of its positive impression. Note too that first class mail is delivered more quickly, is forwarded, and is returned to the sender if undeliverable. Interestingly, stamps on the envelope rather than the postage meter seem to increase the response rate.
The self-addressed stamped envelop is essential, but only use the kind where pay only if the envelope is used [requires postal permit]. This will cost more than first class mail by the piece, but will save money because many of the envelopes will not be used.
Early in the week works best, but avoid Monday because of mail buildup over the weekend. Avoid holidays, December, and early January.
A post card reminder/thank you jogs memories and reminds. Send after one or two weeks. This should make the response seem more important. Text includes reminder of the respondent's importance and an invitation to get a replacement questionnaire if needed.
After the third or fourth week, send a letter and a replacement questionnaire with essentially the same message.
After the sixth week, send a final, certified letter and replacement questionnaire [individual researchers are unlikely to afford this step].
It is important to have a logical, well ordered process to check in responses as they are received and begin data coding.
