Pilot study
ESF: DXH_MET2 Metodologie 2


Recap from DHX_MET1
•So far we have discussed mainly the conceptual side of your projects
•Research question as a focus and a „handle“
•Hypotheses as informed expectations, and clarifications of RQ
•Variables as (observable) representations of the concepts in RQ/H and their measures
•Designs allowing us to use the measures to create meaningful data to answer RQ
•
•
•

Recap
•We have not talked much about
•Data analysis
•
•Practical side of the projects
•There are numerous details in designs-measures-analyses which are necessary to make it work, or
threaten the success of the whole study
•It is easier to learn them hands-on trying to understand them
•

Conceptually, projects are (more or less) in progress…
•… it‘s time to prepare a pilot study
•
•
•
•Plan for this semester: do a small pilot study for one of your RQ
•4 meetings:
•Take stock, see what‘s missing; measures, materials, instructions
•Finalize procedure/protocol from recruitment to debriefing, decide on participants, check ethics
and seek approval
•     (in the meantime) …. brush up data-analysis skills
•Reflect on pilot experience, refine data-analysis plan and power analysis
•

Focus of DHX_MET2
•Projects with people as the primary objects or sources of data


Purpose for a Pilot study
•test feasibility of plan
•methods work
•participants go through the whole procedure without issues
•data look as expected
•analysis feasibility
•do people (participants) function as I imagine them?
•identify weak spots - tune the plan
•
•Murphy‘s laws apply
•
•

•from Leong, Austin (2006)



Choose a RQ/H you want to pilot
•in groups of 2 or 3


Take stock
•


Measures, materials
•Full version of your measures + what it measures
•If some stimuli materials are used, what exatly are they going to be

Instructions, administration
•How exactly are you going to administer the measure to a participant


Gather what you have and show it to a colleague
•„I‘m going to measure the participants‘ CHARACTERISTIC(S) i the following way:….“
•As a colleague look at it from the perspective of
•a participant – how do you react to the measure, the intention to measure..
•a colleague – do you believe it will work as expected?
•Be sceptical, be critical … of the materials

Possible sources of problems in asking questions. Respondents….
•May not understand, may misunderstand
•May not know the answer or how to get to answer
•May not be motivated to invest energy in getting the best (truest) possible answer
•May have trouble fitting their answer to the response scale you offer
•May not want to tell you the (known) answer   (even though they agreed to participate)
•May not even want to know the answer
•May have their own agenda with respect to your study
•May just wanna have fun
•

Possible sources of problems in observing behavior (e.g. games)
•Opportunities for observation error – vague definitions of categories, high cognitive demands on
observer
•Missing important situational factors, determinants
•Fundamental attribution error
•Missing unobservable personal variables
•Not optimal scale of behavior – too micro, too macro
•
•
•
•…. unless the observed behavior does not represent something else
•

Homework
•Finalize measures, materials, instructions for individual measures
•
•
•
•Next, we will combine it with design plans into a protocol for the pilot