Engagment 2: Gearing up for and initiating the pilot study

Pilot study

a trial run of the entire study from start to finish that increases the likelihood of success for the main study.

It is a dry run for the study procedures.

Pilot study considerations include

1. Safety of study procedures

2. Potential for recruiting participants

3. Examination of sequencing and other particular steps

4. Increasing researcher experience

5. Feasibility of study intention

Pilot Study Considerations include:

1. _______ of study procedures

2. _______ for recruiting participants

3. Examination of ________ and other particular steps ex: Randomization; Blinding processes

4. Increasing Researcher _________

5. Feasibility of the study ________.

1. Running quick, preliminary data analysis

2. Revising the research questions

3. Training the researchers

4. Convincing research boards and committees about competence

5. Justifying the relevance of the study

Reasons for Pilot Studies

1. Running quick, preliminary data _________

2. Revising the research _______

3. Training the _______

4. Convincing research boards and committees about _______.

5. Justifying the ______ of the study

1. Results or findings are only based on a small number of people serving as participants in the initial stage

2. Problems or concerns may occur during the actual study execution

3. Contamination issues

4. Requires significant investment of researchers translating to logistical problems to the researchers

5. Creates possible duplication of efforts as may be evaluated by research bodies

Problems with Pilot studies

1. Results or findings are only based on a _____ number of people serving as participants in the initial stage

2. Problems or concerns may occur during the actual ______ execution

3. ________ issues.

4. Requires significant investment of _______ translating to logistical problems to the researchers

5. Creates possible duplication of ______ as may be evaluated by research bodies

publication

Pilot studies are considered as independent studies worthy of _____________.

Publication bias

this refers to the tendency of journal and publishing bodies to select only the researches that have subjective favorable results or findings.

overestimation and underrepresentation

Publication bias is the tendency of journal or publishing bodies to select journals that have subjectively favorable results or findings, which may lead to ________ and ________ of real outcomes.

underdiscussed, underused, underreported

Pilot studies are under______, under_____, and under_________.

instruments

One misnomer about pilot studies is that it pertains to the pre-testing of research _______. In reality, pilot studies encompasses other aspects of the study like sampling, data procedures, etc.

validity and reliability

Another misnomer about pilot studies is that it is commonly equated ONLY with _________ and ________ concerns of research instruments.

learning experiences

Pilot studies offer _____________ that must be documented and shared.

notified

An important ethical point to consider, however, is to clearly explain the CHARACTERISTICS of a pilot study to the participants. In other words, the participants must be _______ that based on the outcomes of the pilot study, the main study may not be performed.

Quantitative

Quick instrument check for ________:

1. Have we obtained the approvals for our instruments in however ways soliciting approvals apply to us?

2. Are these instruments used by other researches?

3. Have the instruments been used to a similar group of people that we intend to study?

4. Do we have validity and reliability citations from other studies that we can cite in our own work?

5. Did the instruments come from scholarly and reputable corpora?

6. Do we have information on how to score and interpret the instruments?

adopting

In _______ instruments, the reliability and validity research studies conducted on those instruments can be used to the current study. 

It is simple and requires little effort. It also saves time in making significant changes.

linked

When using adopted instruments, the current study is ______ to all other studies using the same instruments.

modifications

Adopted instruments may require few _________ IF NECESSARY.

example:

original: i read academic journals before going to sleep.

revised: i read materials before going to sleep.

1. Who developed the instrument

2. Who validated the instrument

3. Other studies that have used the instrument

When using ADOPTED INSTRUMENTS, describe the instruments sufficiently in the instruments section:

1. Who _________ the instrument

2. Who __________ the instrument

3. Other studies that have used the instrument

Adapted 

In _________ instruments, the population by which instruments were created is different with your intended research informants. 

The manifestation of the variable may be different from the group it was created and for the informants you intended

changes 

Adapted instruments require more substantial _________. 

It follows the general design of another instrument but adds items, removes items, and substantially changes the content of each item.

It is ALMOST similar to creating a new instrument 

Changes

When using adapted instruments, the researcher should describe the instruments sufficiently in the instruments section:

1. Who developed the instrument

2. Who validated the instrument

3. Other studies that have used the instrument

4. _______ made in the tool.

Develop

Whenever possible, it is best for an instrument to be ADOPTED. When this is not possible, the next best option is to ADAPT an instrument. However, if there are no other instruments available, then the last option is to _________ an instrument.

Courtesy 

On Academic ________:

"Whether adopting or adapting an instrument, it is always courteous to email the authors of the instrument to ask permission. The authors will rarely deny your request, but its a polite academic courtesy to let another researcher be aware of how their instrument is being used around the world”

Copyright

When using existing instruments the researchers must identify the _________ holder.

Publisher

When using existing instruments the researchers must seek permission from the __________.

contact 

When using existing instruments the researchers must _______ the authors of the instrument.

publicly available

When using existing instruments the researchers must capture the explicit permission to use in case instrument is ________________.

restrictions

Some authors impose __________ for use. Ask permission in case there are major changes that need to be made.

1. Create online forms/surveys 

2. Source/recruit pilot study participants 

3. Administer the questionnaires to pilot study participants following the same study procedures meant for actual respondents

4. Check that all items are answered

5. Record the time taken to complete the tools and decide if it is reasonable. 

6. Ask for feedback to identify ambiguities and for possible difficult questions. 

7. Revise concerned items accordingly

8. Review whether questions still satisfy the study objectives

9. Check for internal consistency

10. Assess the applicability of the tools 

11. Review other participants’ feedback and make the necessary adjustments 

12. Pilot again if needed 

Pilot Study Guidelines (QUANTITATIVE)

1. Create online _____/_______

2. Source/recruit pilot study ______

3. Administer the questionnaires to pilot study participants following the same study ______meant for actual respondents

4. Check that all items are_______

5. Record the ______ taken to complete the tools and decide if it is reasonable. 

6. Ask for feedback to identify __________ and for possible difficult questions. 

7. Revise concerned ______ accordingly

8. Review whether questions still satisfy the study _______

9. Check for internal ______

10. Assess the ________ of the tools 

11. Review other participants’ ________and make the necessary adjustments 

12. ________again if needed 

1. Introduction

2. Informed consent

3. Participant’s agreement

4. Robotfoto/Demographic Sheet

5. Overall instrument reminders

6. Actual instruments (Series)m

7. Debriefing

What are the parts of your google survey?

Introduction

Identify which part of the survey this is:

Thank you very much for your interest to participate and support to our research. We are fourth year BA in Behavioral Science Students who are conducting a pilot study about human resources professionals aged 40 years old and above. Your participation to this pilot study is purely voluntary. The entire survey is expected to be completed within 20 to 30 minutes.

Please be informed that this online google form contains materials that might touch on sensitive issues and may elicit emotional discomfort. Some items of the test included in this study require one to reminisce past experiences which may reopen old emotional wounds. If you are not comfortable, you may choose not to participate in this activity

Participants Agreement

Have a separate section for this part of the survey. It is originally merged with informed consent in the paper and pencil version

answering

For the Robotfoto, check that method of ______ per question has been set correctly (e.g. multiple choice, checklist, short answer, etc.)

instruments proper 

For the Overall Instrument Reminders, mention that the next part will already be the ________________. 

Remind participants to be in a comfortable disposition, to stay relaxed, and answer as honestly as they can. 

instructions

For the Actual Instruments, provide clear and detailed _________

Validity check

After deploying all instruments, include _________

ex:

“I affirm that I have answered all the questions honestly and accurately”

• Yes

• No

“I affirm that I have completely answered all the items in the questionnaire”

• Yes

• No

Debriefing

What is this part called: 

Please help us improve our procedures and study by providing us with brief inputs about the following:

– How do you describe your overall experience in answering this google survey form?

– How long did it take you to complete all the surveys?

– Did you find it difficult to understand any item/s? Which one?

– What suggestions can you provide to help us improve the deployment of this survey in the future?

closing

For the debriefing, put a proper __________ to the google survey and direct concerned participants who may require additional support to the appropriate institutions. 

1. Determine clearly the interview questions

2. Have the initial interview questions reviewed by three experts (faculty or external)

3. Source and select for participants 

4. Prepare a similar google form (parts 1-4 only)

5. Pilot the interview guide

6. Report the modifications made

Pilot study procedures for a QUALITATIVE study:

1. Determine clearly the interview ______

2. Have the initial interview questions reviewed by _______ ________ (faculty or external)

3. Source and select for _______

4. Prepare a similar _________ (parts 1-4 only)

5. Pilot the interview _______

6. Report the _________ made

open-ended

In determining interview questions, ensure that questions are _________.

aide memoir

In determining interview questions, provide bases for the questions included in the _______________.

central question

All interview questions should all relate back to the ____________.

Expert validation

refers to submitting the questions to expert reviewers comprised of the adviser and two other faculty members or practitioners who are knowledgeable in the area.

one to two

Source participants who share as similar criteria as possible to the group actually desired for the main study.

Pilot study participants may involve ____ to _____ individuals.

consent and robotfoto

Upon selecting the participants for the pilot study, prepare the ________ and _______, which are parts 1 to 4 only of the google form.

recording

Don’t forget to mention session __________.

60 minutes; 90 minutes 

According to Jacob and Durgerson (2012), an interview is expected to last for a minimum of __________ up to a maximum of __________.

appropriateness; experience

The purpose of an actual pilot interview is to test __________ of questions and to gain early ________ in doing the task.

rapport-building

The pilot interview begins with ___________.

aide memoir 

The pilot interview entails asking the questions from the ______ ________ one-by-one in a casual manner with the use of probing questions to explore significant areas. 

outcomes

Make sure to note the essential ________ transpiring before, during, and after interview

Debrief

After the interview proper, make sure to ________ the interviewee and discuss the study briefly. Ask how he or she is feeling.

Ensure that the interviewee will leave the meeting feeling comfortable and clarified about what happened.

cool and warm

Report the modification made. Attempt to perform ______ (transcribing and organizing) and ______ (coding, bracketing, categorizing, and thematizing) analyses to check that the process answered study questions

transcribing and organizing

cool analysis

coding, bracketing, categorizing, and thematizing

warm analysis 

aide memoir and study procedures

Revisit the __________ and __________ for re-evaluation.

Evaluating

__________ the PS Outcome:

1. Termination of the study (cannot proceed with the main study due to..)

2. Can proceed with the main study after modifying the study design

3. Not necessary to modify the study design but requires thorough monitoring throughout the study procedures

4. Can proceed without any modifications