How to collect data: questionnaire and sampling

Survey fatigue

The most surveyed group in the society

More likely respondents are

What matters

Which surveys not answered

Students are the most surveyed group in the society -> response rates

are falling

• Some studies have suggested that more likely respondents are:

• females

• ethnic majority

• highly performing

• Topic matters

• Long surveys not answered

• When do you send it out

Sample in quantitative and qualitative

1. Objective of a sample

2. Sample size creteria

3. Ideal sample

4. Generalisation possibilities

Sample error and bias

• If you want to generalize, sample needs to be representative

• Sampling error – the difference between a sample and the population

• Sampling bias – a distortion in the representativeness of the sample that arises when some members of population have small or no chance of being selected in the sample (systematic)

• For example: interested in how performance appraisal affects work motivation (population equally divided)

Increase sample → sample error decreases.

BUT if sample is very biased, increasing the sample is not helpful

Sampling procedures/methods

Simple random sample

• Each unit of population has an equal probability of inclusion in the

sample

• Sampling frame (list of population) is required - usually unrealistic!

• Random numbers, which avoids human bias (pure mechanical

selection)

• Not dependent on availability of a person

Consider how this can be done, if:

• one wants to research how the students perceive the study quality in

Estonian universities

Systematic sample

• Select units directly from sampling frame

• Select a random start number

• Depending on how big is the desired sample, decide on „step“ (every

2nd, 10th, etc)

• If sampling frame is ordered according to something (perhaps position in organization), could be possibly rearranged

Stratified random sample

• If we want the sample to exhibit representation of different criteria

(stratifying by some criteria)

• For example: departments of a company, study levels of university, ...

• We have separate sampling frames, but then proceed with simple random sample or systematic sample

Multi-stage cluster sampling

• If dealing with very dispersed population (country or big city), then

probability sampling would mean a lot of travel (cost)

• Clustering (groupings) can be used

• For example: want a sample of 500 employees who work for 100

largest companies in Estonia

• Simple random sampling to obtain 10 companies (clusters) among those 100

companies (or look also at sector, to allow bigger diversity)

• Simple random sampling to obtain 50 employees in each of the selected 10

companies

Non-probability sampling

• Convenience sampling

• A sample that is available

• For example: want to research business managers, conduct a survey among MBA students (work as managers)

• Useful for piloting or for some preliminary analysis

• Snowball sampling

• Making initial contact with small group of people, who are then used to establish contacts

with others

• Usually used in qualitative research, not quantitative

• Quota sampling

• To produce a sample that reflects a population in terms of the relative proportions of people in different categories (gender, ethnicity, age groups, ...)

• Not done randomly, but decided by interviewer

• In case of all non-probability sampling: issues of generalization!

Open questions

Typically qualitative (unstructured)

Adv:

1. Unconventional (out-of box) answers

2. What knows about the topic

3. To research new topic or to get new knowledge

Disadv:

1. Answering takes more time

2. Data analysis takes more time (coding)

3. If several researchers, the interpretation might differ

Closed questions

Typically quantitative research (structured approach)

Advantages:

• Simple to answer and analyse

• Answers are comparable, enable to conduct statistical tests

• Response options can help to understand the meaning of the question

Disadvantages:

• No spontaneity (to alleviate it, use the option „other“)

• Can be difficult to define response options that exclude each other

• Can be difficult to provide all possible respsonse options (the list can be very long)

• Respondents might intrepret the question differently

• Respondent might get angry, when the response option that (s)he wants to use is

missing

Golden rules for questions

• Avoid terms of several meanings

• For example, „often“ or „regularly“ can be differently interpreted; „x times

per day/week...“ might be better versison

• Avoid long questions

• Avoid double-barrelled questions (which include several questions/parts)

• Avoid very general questions (For example: How satisfied are you with studies?)

• Avoid leading questions

• Avoid technical terms (which are not common knowledge)

Likert-scale questions

• Typical closed questions representing ordinal data type

• Originally had 5 response options, but very often also 7 or 9 options are used

• Sometimes, if the indifference is not desired, even number of options are used (for example 4 or 6)

• It is important that:

— The distance between answers is the same

— Symmetrical to the centre (middle answer)

— Consider whether to provide options „Don’t know“ or „Don’t wish to answer“ (if so, definitely outside the „valid“ categories)

Additional suggestions

• Accompanying letter is very important (who is conducting the research,

why, contact)

• Questionnaire must be piloted before actual research

• If necessary, explain the definitions or terms you use in questionnaire

• Think through, can you use single question or need different questions to

measure some phenomen (work with literature)

• If possible, use questionnaires or specific questions that are developed and

tested by other people -> „validated instrument“ (measurement validity)

• Use question banks or statistical classificators for building up response

options