Data Collection Survey Jan Osička Outline • Online survey: theory and practice • Sources of data • Triangulation Survey What is survey and when we use it? Designing and implementing a survey 1) A systematic process of 2) gathering information on a specific topic by 3) asking questions of individuals and then 4) generalizing the results to the groups represented by the respondents. The process • The research aims – what are they? • Does the survey method help the best to achieve them? What are the goals of survey? The process • The research aims – what are they? • Does the survey method help the best to achieve them? What are the goals of survey? • Learn about/describe a population (stakeholders, decision makers, technology users) • Make comparisons between groups (similar groups in different environments, users x non-users) The process Defining the population • What is the target population? • Is it possible to survey all the population? What if not? Timing • When is the data needed? • When is the best time to conduct the survey? • When is the best time to contact the respondents? Self-administered surveys (i.e. online) Pros: Cheap, fast, powerful Cons: • Response rate (30-40%) • No control over the environment • Survey appearance depends on platform and browser (reliability risk) • Variation in internet usage, mail reading habits The importance of context (the purpose, use of data, anonymity) and first contact. Self-administered surveys (i.e. online) Design considerations: • Unobtrusive layout • Intuitive launching and moving through • Email and webpage headers create the first impressions • Contact information necessary Developing questions Question = a measuring device for things that are not directly observable • Reliability (the extent to which repeatedly measuring the same property produces the same result) • Validity (the extent to which a survey question measures the property it is supposed to measure) Ideal question therefore... • Measures the underlying concept it is intended to tap • Does not measure other concepts • Means the same thing to all respondents Types of questions • Open-ended vs. closed ended questions • Rating vs. ranking • Semantic differentials • Question batteries • Contingency questions Rules for asking questions • Avoid technical terms and jargon • Avoid vague or imprecise terms • Define things very specifically • Avoid complex sentences • Provide reference frames • Make sure scales are ordinal • Avoid double-barreled questions • Answer choices should anticipate all possibilities • If you want a single answer, make sure your answer choices are unique and include all possible responses • Avoid questions using leading, emotional, or evocative language Structuring questions within survey • Begin with questions that reflect the announced subject of the study, catch the respondent’s attention, and are easy to answer • Group items that are similar in topic, then group items within the topic that have similar response options • Place personal and demographic questions at the end of the survey Do Do not • Give clear instructions • Keep question structure simple • Ask one question at a time • Maintain a parallel structure for all questions • Define terms before asking the question • Be explicit about the period of time being referenced by the question • Provide a list of acceptable responses to closed questions • Ensure that response categories are both exhaustive and mutually exclusive • Label response categories with words rather than numbers • Ask for number of occurrences, rather than providing response categories such as often, seldom, never • Save personal and demographic questions for the end of the survey • Use abbreviations, contractions or symbols • Mix different words for the same concept • Use “loaded” words or phrases • Combine multiple response dimensions in the same question • Bounce around between topics or time periods • Insert unnecessary graphics or mix many font styles and sizes Testing and training • Internal testing and tweaking (consistency, allignment with the aims, grammar) • Testing on a sample of target population or of a similar population • Do people understand the terms? • Do people complete the survey as intended? Or do they drop out before completing it? • Are certain questions regularly skipped or show no variation? • Does the survey launch properly and work as expected with different browsers? Sources of data Sources of data • Original data: documents, data sets, statistics, interviews • Interpretations: studies, analyses, research papers, presentations Problems with data • Missing pieces and limited access • Compatibility • Money and resources Need for combination of sources Validity and reliability issues Triangulation • Using multiple perspectives to boost validity • Does not aim to arrive at consistency across data or approaches • Reveals inconsistencies to improve the research Triangulation • Data (stakeholders‘ points of view) • Investigator (data/evaluation) • Theory (multiple perspectives/disciplines) • Methodology (multiple methods) • Environment (intervening variables) Triangulation Advantages • Increasing confidence in research data • Creating innovative ways of understanding a phenomenon • Revealing unique findings • Challenging or integrating theories • Providing a clearer understanding of the problem Triangulation Disdvantages • ? Triangulation Disadvantages • Time and money • Investigator bias • Conflicting theoretical frameworks • Lack of understanding about why particular strategies were used Triangulation in energy-related (social) research Theory • Interdisciplinary studies (economics, sciences, politics) Environment • Institutions in different countries Data • Statistics + secondary data Method • Statistical methods + interviews/surveys • Case study + interviews/surveys