Questionnaires and Scales
Questionnaires and Scales
Learning Outcomes
Design questionnaires and scales for different research projects.
Discuss and explain the issues of validity and reliability in relation to questionnaire and scale design.
The aim of this module is to explain the issue of questionnaires and scales in data gathering.
Questionnaires and scales are structured means of data gathering and they are used a great deal in quantitative research.
Quantitative research produces quantitative data, which are numerical data or data that can readily be converted into numerical form.
Qualitative data can be gathered with questionnaires, through the use of open questions.
This explains how to create questionnaires and scales, and when and how to use questionnaires and scales in a research project.
Introduction
Questionnaires and scales are precisely structured data gathering instruments that are specially designed, highly structured, very focused and usually short.
They are widely used in survey research.
They are used primarily in quantitative research to generate quantitative data, although qualitative data can be generated by questionnaires, through the use of open questions.
Quantitative data is factual data.
A survey research methodology works well in large scale research projects.
Surveys are often used in research conducted with large/or geographically spread populations.
Questionnaires can be easily posted or emailed to such populations, or placed on the internet to allow for easy access globally.
Questionnaires and Scales
A survey research methodology is an appropriate research methodology in a large population research project.
Questionnaires and/or scales are the appropriate gathering instruments.
They are structured to ensure that each respondent is asked the same simple, clear, concise and precise questions, and to ensure that the responses made to those questions/issues are also simple, clear, concise and precise.
The data gathered by researchers engaged in very large studies tends to be mostly or entirely quantitative data.
Quantitative data are precise and concise, while qualitative data, on the other hand, tends to be complex, dense, and often quite voluminous.
Questionnaires and Scales - Likert Scale
The sample scale below is an example of a Likert scale.
Likert scale - measuring brand loyalty
1 I am loyal to one particular brand.
2 I am not concerned so much about price; I would buy my brand regardless of cost.
3 If my local shop did not stock my brand, I would travel in order to be able to buy my brand.
Likert scale is named after the person who developed it: Rensis Likert. It is widely used in the measurement of attitudes.
The Likert scale is useful in that as well as measuring the direction of attitudes, it also measures the force of the attitudes.
A Likert scale can be a three-point scale, or a five-point scale or a seven-point scale.
Questionnaires and Scales - Rating Scale
Table below depicts a rating scale.
Rating scale employee Health and Safety training programme
1 Please rate the content of the training programme.
2 Please rate the delivery of the training programme.
3 Please rate the relevance of the training programme.
4 Please rate the communication style of the presenter of the training programme.
A simple rating of any phenomenon can be a useful data gathering exercise. Using a rating scale, the researcher can ask participants in the research project to rate the phenomenon under investigation.
The researcher can ask the participants to rate as many or as few aspects of elements of the phenomenon under investigation as is useful for the purposes of the research.
Questionnaires and Scales - Semantic Differential Scale
Another useful scale is the semantic differential scale.
How do you feel about our new product? Please mark the point on the scale that best represents your opinion.
Unpleasant - Pleasant
Ineffective - Effective
Unreliable - Reliable
Difficult - Easy
Weak - Strong
A semantic differential scale uses opposite adjectives and asks respondents to indicate which of the adjectives best describes the phenomenon under investigation.
Questionnaires and Scales - Bogardus Social Distance Scale
Another scale is the Bogardus social distance scale, developed by psychologist E.S. Bogardus.
Bogardus social distance scale: Measuring prejudice against (in this case) Irish people
I would admit an Irish person….
Into my family by marriage
Into my club as a personal friend
To my street as a neighbor
To employment in my occupation
To employment in my workplace
As a colleague of mine in my workplace
As a visitor only to my workplace
Would exclude from my workplace
In using Bogardus social distance scale, participants in the research simply indicate, by ticking or not ticking the boxes on the right of the scale, the degree to which they would be prejudiced against the population of the study.
Designing Questionnaires and Scales
When you have collected a number of questionnaires and scales, examine them.
Study the way in which they are presented in terms of both organization and aesthetics.
Study the structure and the sequence of questions and items.
Take note of which questions are asked first, and which questions came later.
Study the words, concepts and language used.
Examine the manner in which items and questions are presented.
Take note of the manner in which the respondent was required to respond to the questions and items, ticking boxes, inputting words/or numbers, writing phrases and/or sentences.
Note the structure and presentation of the closed questions, and the way in which the respondent is required to respond to them.
Note the structure and presentation of the open questions, and the way in which the respondent is required to respond to them.
Note all of the different question formats in the questionnaires and scales you have gathered.
Note skips and filters (Devices used in questionnaires to allow respondents to skip over questions that do not relate to them and the way in which they are used.
Note the length of the questionnaires and scales.
Designing Questionnaires and Scales - Key Issues
Key issues when designing questionnaires:
The content of the questions.
The construction and presentation of each of the questions.
The order of the questions.
The length of the questionnaire.
The best guide to the design of an appropriate question or item, to begin with, is the conceptual framework of your research project.
The conceptual framework of the research project is contained in the research question or statement of your project, and it is the question or statement driving your research.
When you begin to design your research, you look at your research statement/question and objectives of your research. Then consider the literature you have reviewed for the research project.
Try to imagine the kind of response the person might make to the research question. This is a useful exercise when attempting to generate research questions.
When you have done this for a series of questions, examine the likely responses to see how well they fit with the data requirements of the research. This is an exercise in critical reflection and also called questionnaire validation.
Practicalities of Asking a Question
The presentation of the questionnaire should be simple, succinct (brief and clearly explained), and professional. The spacing of the questions should be logical and aesthetically pleasing. In other words, the questionnaire should be attractive.
Issues in designing questions and items for questionnaires and scales
Leading response - the possibility of leading participants to a particular response through the way in which the question is framed.
The problem is that the researcher has contaminated the research by prompting respondents through the use of particular brand names.
Ambiguous questions - These are questions which have an unclear meaning, or questions that can have more than one possible meaning.
Complicated questions - Questions should be simple, clear, concise and precise.
Asking two questions, or more, in one question - Always check that you have not done this when you are compiling a questionnaire, and if you have, rephrase your question, turning that one question into two or more questions.
Potentially embarrassing questions and/or questions which you have no right to ask - If you need to ask potentially sensitive questions, such as how much money the respondent earns, or how old the respondent is, it is helpful to aggregate the possible answers and then ask the respondent to indicate which range they belong to, rather than respond with a precise figure.
Asking unnecessary question - Ensure that your questionnaire, or schedule of questions and/or issues to be explored, is succinct and relevant, and absolutely to the point.
Loaded words - Loaded words are words with particularly strong emotional impact.
Unclear or vague concepts - Concepts that are not clearly defined are problematic.
Insulting words - Some words and phrases are simply insulting or even degrading. It is important to remember that a word or phrase that is perfectly acceptable in one culture may be quite unacceptable in another.
Humour - As a rule, it is best to avoid humour completely. Unless humour is the topic of your research. Perceptions of what is funny vary from person to person as well as from culture to culture. As this is the case, it is best to avoid it completely.
Slang and colloquialism - A simple formal language should be used throughout. Avoid slang and colloquial words and expressions.
Ethnocentrism - Ethnocentrism means viewing the world, and in this context consequently organizing and designing data collection for your research, from your own ethnic or cultural perspective.
Sexist, racist, ageist or disablist language - Ensure that you are not, through the language you use, discriminatory towards or displaying prejudice against any class, race or gender; younger people or older people; or people with different levels of physical and/or intellectual abilities.
Abbreviations and jargon - Do not use abbreviations or jargon. Make each question/item as simple and clear as possible.
Respondent bias - Acquiescence bias, social desirability bias and prestige bias.
Talking down respondents - Do not 'talk down' respondents, by treating them disrespectfully or by patronizing them.
Double negatives - Double negatives in a question or sentence can be confusing.
Avoid negative questions.
Ask simple, easy-to-answer questions - Ask questions that the respondent will be easily able to answer. Don't ask too much for respondents; keep to a minimum the effort they will have to make in order to respond to the questionnaire.
Give clear instructions - Ensure that you give clear instructions to respondents. Make sure that respondents have all the instructions, direction and guidance they need in order to be able to properly and fully respond.
A Pilot Study
A pilot study is a test of the design of the research project, or a test of the data gathering instrument(s) designed for the research.
In general, all data gathering methods should be tested, and in your reading for the literature review and for the research methods element of your research, you will notice that almost every data collection method you come across will be subjected to a pilot study.
A pilot study is carried out using a small number of respondents.
These respondents should be similar to the actual respondents in the study, but they should not be respondents in the study.
Usually, pilot studies are carried out with 5-15 respondents, depending on the size of the study.
Any issues that the pilot study throws up can be dealt with before the real study takes place.
Response Rates
A response rate in research is a count of the number of valid responses received to a data gathering exercise.
The higher the response rate the better.
In F2F data gathering exercises, it is not unusual to achieve 100% response rates, in telephone interviews it is not unusual to achieve 80% response rates. In postal surveys and in online surveys, response rates are often considerably lower than this.
Non-responses change the nature of the study and the claims that can be made about the study.
If everyone responds, you can apply your findings confidently to the population of the study. The fewer the number of valid responses, the less confident you can be.
Response Rates - Improving Responses
There are a number of ways through which responses can be improved.
Respondents can be contacted and encouraged to respond.
Incentives can be given in order to improve the response rates.
Clear and persuasive information sheets and informed consent forms can improve response rates
More people can be encouraged to respond by clarity, simplicity, and brevity in the design of the data gathering instrument. Courtesy also will improve response rates.