Lecture 4 Creating data by asking questions DHX_MET1 Methodology 1 Stanislav Ježek Faculty of Social Studies MU How to capture phenomena we want to study •In early exploratory research we gather broad data informing us about the phenomenon, associated phenomena, the context … • •When we have sufficient theoretical framework indicating which characteristcs of phenomena (=variables) are relevant we are trying to „measure“ the variables Broad definition of „measurement“ •Gathering of data allowing us to make inferences about the value(s) of researched variable(s) •continuous or categorical numeric •qualitative •Operationalization of variables •To be differentiated from measurement proper of quantities • •In most social-science research we measure (place) variables on •nominal scales – unordered categories •ordinal scales – categories ordered by the intensity/quantity of the variable •interval/ratio scales – closest to quantitative scales with a unit of measurement •a number of theoretical problems, but pragmatically it has its heuristic value • Methods, methods of measurement, measures – procedures for generating data •In line with continuity and reproducibility of science we tend to use methods used in previous research •How to choose an appropriate method? 1.In previous research (see also protocol, paradigm) 2.In special books 3.Sometimes we develop our own Practically – what constitutes a method? •Materials, tools, instruments – questionnaire, observation schedule, stimulus material, measurement device/instrument •Administration procedure – how to use the above to get data •Coding/Scoring procedure – how to convert raw data to variable values •Interpretation guidelines – what the variable values mean/represent •In applied psychology, also norms describing the distribution of values in a relevant population • • • • • How to choose a method? •Quality of measurement •measurement validity and reliability • •Comparability with previous research • •Practicalities •time requirements •skill requirements •financial and other resource requirements •availability Quality of measurement •Guided by 2 basic questions: 1.Is our measure systematically affected only by the variable we want to measure? •VALIDITY 1. •2. To what extent is our measure affected by random effects? • RELIABILITY • ARGUMENTS for VALIDITY •Theoretical argument – content validation •Empirical arguments •correlation with similar measures – criterial, convergent validity •non-correlation with meaure of a related but theoreticaly independent variable – discriminant validity •Never-ending process of validation! SOURCES of (UN)RELIABILITY •Reliability – precision, accuracy, absence of random noise in values • •STABILITY – correlation of repeated measurement •OBJECTIVITY •agreement of the results of administration+scoring/coding by different users •quality of standardization •intra-class correlations, kappas •INTERNAL CONSISTENCE •many measures try to assess a variable by a number of similar attempts – questions, tasks – generally „items“ •All items should provide consistent, unconflicting information about the value of the measured variable •Cronbach‘s alpha … McDonald‘s omega • •OBSERVATIONAL •OBSERVATION •PRODUCT ANALYSIS •TRACES, USAGE DATA •INSTRUMENTS •EXPERIMENTAL PROTOCOLS •PSYCH. TESTS •ASKING QUESTIONS •INTERVIEW • •QUESTIONNAIRE •RATING SCALES • •ATTITUDE SCALES • 2 TYPES of METODS ASKING QUESTIONS - INTERVIEW •It is basically MEDIATED OBSERVATION • •Interview is a conversation with a specific PURPOSE – CREATE DATA. •Interviewing is a SKILL. • • •BASIC INTERVIEWING SKILLS •MANAGEMENT to steer it towards its purpose •ASKING questions (wording, ordering, asking them) •LISTENING to answers (+ taking notes) • CÍL je třeba mít jasný Zastavit se u záznamu – získat souhlas a natáčet. Spoléhání na záznam může vést k „neposlouchání“. The level of STRUCTURE •To what extent is the course of the interview predetermined, standardized – structured? • •UNSTRUCTURED – What do you think of euthanasia? •Open, broad questions mapping the interviewee‘s representation of what we want to learn about. •Intervier facilitates production, clarifies, and steers towards the topic •STRUCTURED – Do you think euthanasia should be legalised or not? •Open or closed questions focusing on narrow aspects – variables •Limited, often prepared clarification and facilitation options •SEMI-STRUCTURED – What do you think abou the legalisation of euthanasia? Under what conditions is euthanasia possible ? •A series of topics covering research question •Like Unstructured, but some ideas are introduced via a question even if the interviewee does not mention them • •… INTERVIEW SCHEDULE Strukturovanost= neměnnost otázek neměnnost způsobu (pořadí) jejich kladení --- MALÁ INTERAKTIVNOST (spíše) uzavřenost otázek Absence přizpůsobivosti průběhu rozhovoru Vysoká přesnost a opakovatelnost PHASES OF INTERVIEW •INTRODUCTION – interviewer, pupose and topic, confidentiality, recording agreement, rapport •WARM UP– simple, non-threatening questions •THE BODY OF THE INTERVIEW •COOL-OFF – deal with built up tension or questions •CLOSURE – thanks, debriefing, faerwell. „Hand on the doorknob“. To vše dohromady je v rozhovorovém schématu. INTERVIEW MANAGEMENT §Have it WELL-REHEARSED §PAY ATTENTION, LISTEN, and let it show §NEUTRALITY §Ask calmly, without implying emotions, positive or negative §Minimize LEADINGNESS, SUGESTIVENESS. Do not indicate what you‘d like to hear §Order questions from the general to the more specific (funnel) §ENJOY, BE INTERESTED … otherwise – INTERVIEWER BIAS PAYING ATTENTION & PROBES •Techniques maintaining rapport - listening, response clarifications, probes facilitating further responses. • •GENERAL PROBES •keeping short silence (krátké) •questioning look •„mmmmmmm“ •repeating response, paraphrasing, summarising •nodding (verbal & non-verbal) •SPECIFIC PROBES •Prepared clarification questions, part of interview schedule Ukázat příklad na klima. QUESTION FORMATS •OPEN-ENDED •Most laborious for respondents •Most open (sometimes not true) • •CLOSED •facilitating the understanding of a question and responding • •Multiple-choice •Rating scales – direct quantification • • Rozdíl mezi otevřenými/uzavřenými a těmi speciálními spočívá také v míře QUESTION WORDING TIPS •RELEVANCE/FAMILIARITY r. should be easily able to produce the answer •SIMPLICITY / ELABORATION •ASK ONE THING (x DOUBLE-BARREL), beware of conditions in questions, •EASY LANGUAGE, avoid slang, vague, difficult words •AVOID NEGATIVES especially double-negatives •INCLUDE EXPLICITLY BOTH POLES •offer „DON‘T KNOW“, anytime it is a plausible response •BEWARE of CONTEXT created by previous questions Aby usnadňovala vybavení těch paměťových stop, které chceme a minimalizovala nežádoucí „asociace“. 4 PHASES OF RESPONDING TO QUESTIONS COMPREHENSION Identify what information si needed Purpose included RECALL FROM MEMORY JUDGEMENT Developing an answer from recalled material RESPONDING Mapping onto a response scale. Decision to produce to response. Tourangeau, Rips, Rasinski (2000) QUESTIONNAIRE •Questionnaire is an interview standardized so well that it can be self-administered • •All issues with comprehension etc. must be sorted out through PRETESTING • •Modes of administration •oral/personal, CAPI •pen and paper •phone, CATI, Skype •online • • SUMMARY •Quality of measurement – validity and reliability •Interviewing as eliciting observations •Four-phase response process as a cognitive theory of question wording, ordering and other effects •Questionnaire as a highly-structured interview allowing self-administration Reading •Chapters 7 and 9 plus this presentation. • • • •What‘s next? •Direct observation and measurement.