Two Central Research Frameworks in Information Retrieval: Drifting outside the Cave of the Laboratory Framework Peter Ingwersen* & Kalervo Järvelin** * Royal School of LIS, Denmark (pi@iva.dk) – and Oslo University College, Norway ** Tampere University, Finland (kalervo.jarvelin@uta.fi) 2011 2 Peter Ingwersen Agenda n1. Introduction – Wilson Onion Model n2. The Laboratory Framework for IR. nProvides system-driven evaluation nThe Framework – trapped in the Laboratory Cave nDrifting outside the Lab. Cave towards Context: n3. Alternative ISR models leading to: nUser-driven evaluation – and n4. The comprehensive cognitive framework for research on Interactive IR nContexts – Relevance – Interaction 2011 Peter Ingwersen 3 Information behaviour and IR n n n n n n n T. Wilson´s Onion Model, 1999 - extended: Seeking IR Job-related Work Tasks Interests Non-job-related Tasks and Interests Daily-life behavior Information behaviour Interactive IR Behaviour 2011 Peter Ingwersen 4 Information behaviour … and other central concepts in Information Studies nInformation behaviour: nto create information – e.g., on the Net - blogs; human indexing, inclusing social tags; nto produce publications – e.g., as publisher nto communicate – face-to-face; chat; e-mail nto manage information sources – e.g. KM; selectivity nInformation seeking (behaviour) nInformation behaviour with desire for Information nInformation need exist – even mudled nSearching information sources – e.g. colleagues nInformation Retrieval (I)IR nSearching information space via systems – Digital Library & Assets (interactive IR) nRetrieval models; relevance feedback & ranking; query modification; auto indexing and weighting; 2011 5 Peter Ingwersen The Laboratory Framework for IR nOne simplistic and robust framework nSearchers not present nMany competing retrieval models under one framework nFew and well-defined variables nAlmost full control of experiments n n 2011 6 Peter Ingwersen The Laboratory Approach to IR This is information retrieval, isn’t it? But where is the lab? Docu- ments Represen- tation Database Search request Query Matching Represen- tation Query Result Pseudo Relevance feedback The Lab Included into a Framework Docu- ments Represen- tation Database Search request Query Matching Represen- tation Query Result Evaluation Result Evaluation Relevance assessment Recall base Pseudo Relevance feedback The Lab IR Cave in Context Docu- ments Represen- tation Database Search request Query Matching Represen- tation Query Result Evaluation Result Evaluation Relevance assessment Recall base Context The Lab IR Cave, with a Visitor Docu- ments Represen- tation Database Search request Query Matching Represen- tation Query Result Evaluation Result Evaluation Relevance assessment Recall base Context 2011 10 Peter Ingwersen LabIR: The Framework nSearchers, “users”, lost nHave no interesting explicable attributes (all-alike) nBut nevertheless hiding in the relevance ssessments: nRelevance assessments are rarely seen as problematic nOnly related to the requests and documents nThe independence assumption nVariations neutralized statistically nInteraction: nExcluded: interface, searchers, search/seeking process nRegarded as a sequence of simple independent topical interactions; no saturation nOnly 1-2 runs allowed (at least with Rel. Feedback in probabilistic model: … user-driven?!) nMotivations: nFramework for the (algorithmic) IR phenomenon and IR system evaluation to support system design. Much of Lab IR research disregards searchers. However, “users”, while having no interesting explicable attributes, are nevertheless hiding in the relevance assessments as hidden variables. Relevance assessments are rarely seen as problematic nor essentially related to anything else than the requests and documents. Interactive Lab IR research takes searchers into account (e.g. the TREC Interactive Track; Hersh & Over, 2000). However, in these efforts, searchers are made to find (through a given a system) documents for given static topics that someone else assessed as topically relevant. Recall Base and Evaluation. The recall base is derived by extensive pooling of possibly relevant documents for each topic. The participants of test collection construction retrieve, for each topic, documents and the Top-n (n being, say 100) results are merged and then assessed by independent assessors typically using binary relevance and very liberal relevance criteria (Sormunen, 2002). Each pool is assumed to contain all relevant documents for a topic. Evaluation may use various metrics, typically based on recall and precision, e.g., MAP across a topic set. 2011 11 Peter Ingwersen LabIR: The Characteristics – 2 nDocuments & Rep nUnstructured natural language news items - ‘just stuff’ nindependent indexing features nRequests & Queries nUnstructured natural language word bags; one, verbose & static i-need version n‘just stuff’ nMatching and Results nMatching based on document and requests representations as guided by a retrieval model nResults typically ranked lists of document reps; list items have rank, score and binary relevance (posteriori assessments) In real life, interactive searchers often use multiple queries through reformulation until they are satisfied or give up. Experimental set-ups assuming one query per topic are insufficient in multiple query session evaluation, where the searcher’s reformulation effort matters. Initial queries may fail due to the long and widely known vocabulary problem – the problem of formulating a good query. Even true domain professionals have different interpretations, and consequently, construct differently behaving queries – even when facing the same situation. This does not become apparent when (verbose) topics of test collections are (automatically) used in experimentation. In real life there seldom are lengthy topical need descriptions – there rather is a multitude of possible interpretations that need to be mapped to a collection. 2011 12 Peter Ingwersen Nested Framework … drifting into Contexts outside the Cave Docs Repr DB Request Query Match Repr Result Work Task Seeking Task Seeking Process Work Process Task Result Seeking Result Work task context Seeking context IR context Socio-organizational& cultural context Here one cannot avoid taking the user seriously and duly into account. The traditionnal laboratory evaluation metrics are insufficient. One needs to evaluate the IR system’s contribution to information seeking or task performance. 2011 13 Peter Ingwersen The Integrated Cognitive Research Framework for IS&R– its basic model The Lab. Framework 2011 Peter Ingwersen 14 Basic IR research approaches - 2 nUser oriented approach - 1970s... nIn operational settings (Boolean(like) systems) nScientific/technical information as object nWith users and often real information needs nInformation needs: variable over session time nWork and search tasks (reasons) not considered nRelevance assessments: by the users themselves nIntermediary-end user interaction & behavior nOrg., social or cultural context rarely involved nMeasures: Recall & Precision; Satisfaction 2011 Peter Ingwersen 15 Information seeking studies in relation to user oriented and cognitive IR nCommonly highly communication oriented nWork tasks and system features rarely included nIS Theory foundation: T. Wilson (1981); from 1986: Dervin & Nilan – but also Talja & Savolainen (2000) nThere are exceptions who moved into cognitive IS&R (Tom Wilson; Kal. Järvelin; Pertti Vakkari; Tefko Saracevic; Amanda Spink; Peiling Wang) n 2011 Peter Ingwersen 16 Wilson´s 1981 model of Information seeking 2011 Peter Ingwersen 17 Dervin & Nilan´s sense-making (1986) – (The Turn, p. 60) 2011 Peter Ingwersen 18 Carol Kuhlthau’s stage model – 1991/94 - (The Turn, p. 65) 2011 Peter Ingwersen 19 IS and (I)IR into IS&R nByström/Järvelin, 1995 – IS&R model nSaracevic, 1996 – stratified model nIngwersen, 1996 – including contextuality nWang & Soergel, 1998 – assessing the retrieved/found document nVakkari, 2000 – IS into IS&R – model n nModels become increasingly comprehensive and generalized to cover IR components too 2011 Peter Ingwersen 20 IS&R model, 1995: Bystöm & Järvelin, fig. 2 (From: The Turn, p. 69) 2011 Peter Ingwersen 21 Saracevic´ stratified model for IIR (1996) 2011 Peter Ingwersen 22 Wang & Soergel 1998 Type Abstract Document Values Criteria DIEs Decision processing combining deciding Knowledge of topic person organization journal document type Decision Rules Elimination Multiple criteria Dominance Scarcity Satisfice Chain Author Title Orientation Topicality Date Series Journal Relation Authority Availability Novelty Quality Emotional Social Conditional Functional Epistemic Rejection Maybe Acceptance DIEs: Document Information Elements Values: Document Values/Worth (From: The Turn, p. 201) 2011 Peter Ingwersen 23 IR and relevance in Seeking context – Seeking into IS&R: Vakkari 2000 2011 Peter Ingwersen 24 Task-based IS&R nOriginates from Järvelin (1986) Ingwersen (1992) and developed empirically by Byström & Järvelin (1995) and Vakkari (2000; 2001), etc. nTask complexity is one of several characteristics of work/search tasks to be investigated nLeads to Work task simulations (cover stories) in IS&R investigations (Borlund Package, 2000 …) n 2011 Peter Ingwersen 25 Situation in context > Work task > Perception > Uncertainty > Information Need nThe more complex the situation and work task - the greater the uncertainty and knowledge gap (Byström & Järvelin, 1995); nThe information need becomes increasingly ill-defined – people become knowledge sources nRecently in Lab. IR: Situational (task) impact on search behaviour – relevance assessments: systems design should support cognition 2011 Peter Ingwersen 26 Simplistic model of ISR – short-term interaction – in context Information objects IT: Engines Logics Algorithms Interface Information Seeker(s) Org. Cultural R Query R = Request / Relevance feedback Short-term IS&R & social interaction Cognitive transformations and influence over time Modification Social Interaction Social Tagging Recommender techniques Social Context 2011 Peter Ingwersen 27 Ingwersen Central Components of Interactive IR – the basic Integrated Framework The Lab./DL Framework In situ recommendation In situ tagging 2011 28 Peter Ingwersen Central differences between the Lab. and integrated cognitive frameworks nConception of Information – and hence: nConception of Relevance nTask dependency (in Cognitive Framework) nIR System Setting – also seen as context to actors nRole of Interaction – the central issue nRole of Intermediary – interface issues (not in Lab.) nContext characteristics nEvaluation Approaches nIntegrated perspective of all actors, processes and outcomes n 2011 29 Peter Ingwersen The Integrated Cognitive Research Framework for IS&R– its basic model The Lab. Framework 2011 30 Peter Ingwersen The applications of the Model & the Cognitive Framework nIllustrating the roles of actors in a variety of cases of information behavior, like IR interaction; nPointing to core components and information processes depending on (or influencing) such cases – i.e., nPointing to kinds of context – next slide; nPointing out central variables involved in a variety of research designs – with a number of independent variables 2011 31 Peter Ingwersen Cognitive Framework and Relevance Criteria [USEMAP] Docs Repr DB Request Query Match Repr Result A: Recall, precision, efficiency, quality of information/process B: Usability, quality of information/process C: Quality of info & work process/result Work Task Seeking Task Seeking Process Work Process Task Result Seeking Result Evaluation Criteria: Work task context Seeking context IR context Socio-organizational& cultural context D: Socio-cognitive relevance; social utility; quality of work task result 2011 32 Peter Ingwersen Relevance and Evaluation nSome information more relevant than other nRelevance changes over time nMajor (horizontal in model) types of relevance: nAlgorithmic / System relevance (objective) nTopical (aboutness interpretation) nPertinence (information need satisfaction – isness – authority of sources – novelty – currency) nSituational (usefulness of objects to task/interest: refs.) nSocio-cognitive/social utility (group interpretation of objects – also over time: citations – recommender systems /collaborative filtering – web inlinks)(evidence exists) nNB: Emotional (associated with all subjective higher order relevance types) n Higher Order 2011 33 Peter Ingwersen The Integrated Cognitive Research Framework suggests nApplications of research designs nComparisons of retrieval (and seeking) in different types of collections nComparisons of experts and novices and other actor types by features nComparisons of simulated task types (degree of manipulation and semantic openness) – or real tasks – for experimental control nConsequences for IR performance 2011 34 Peter Ingwersen The Integrated Cognitive Research Framework informs about … nCentral variables to combine as independent ones nVariables to be kept controlled in a setting nWhat kind of variables that are hidden! nDependent variables depend on the research goals nNovel possible research designs, settings and measures … there is a lot to do - really! THANK YOU!