Empirical research methods reported in high-profile LIS journal literature
Philip Hider ⁎, Bob Pymm
School of Information Studies, Charles Sturt University, Boorooma Street, Locked Bag 675, Wagga Wagga, NSW 2678, Australia
A B S T R A C TA R T I C L E I N F O
This study describes a content analysis aimed at identifying the distribution of empirical research strategies
and techniques reported in high-profile LIS journal literature published in 2005. For each article, researchers
identified the overall research strategy, the data collection technique, and whether the type of analysis was
quantitative or qualitative. The taxonomies used in the analysis were those based on Järvelin and Vakkari's
[Järvelin, K., & Vakkari, P. (1990). Content analysis of research articles in library and information science.
Library & Information Science Research, 12, 395-422] study in order to compare the results with the earlier
findings derived from the 1975 and 1985 literature. The survey approach remains the predominant research
strategy in both library science and information science. However, there was a marked increase in
experimentation. There were more modest increases in the use of qualitative approaches, except for
historical research, which showed a marked decline. This study's findings will inform development of
methods courses in doctoral programs, which aim to cover the most commonly used strategies and
techniques in contemporary LIS research. Revised taxonomies, which include previously unspecified
strategies and techniques such as ethnography and transaction log analysis, are suggested for future content
analyses of LIS research.
© 2008 Elsevier Inc. All rights reserved.
1. Introduction
Recent surveys in the United Kingdom (Morris, 2006), the United
States, and Korea (Park, 2003, 2004) found that library and information
science (LIS) departments offered a wide range of research methods
courses. Many of the courses were not compulsory, and they
emphasized different methodologies and methods. Among other
things, this variation may reflect various research interests and expertise
across LIS departments, as well as different topics and approaches
of research students, at a particular point in time. (In contrast, Park
(2004) found a lack of correspondence with research students' methods
in some US courses). It is obviously important for research
methods courses, particularly in doctoral programs, to cover in depth
those methods likely to be appropriate for the research proposed by
the students taking the courses. It is also important for syllabi to cover
a broad range of methods so that students may select appropriate
methods. This also gives students some grounding in methods that
they may wish to use in the future, perhaps outside of a particular LIS
department. A methods course in a doctoral program that focuses too
acutely on a particular set of strategies and techniques, at the expense
of other commonly used approaches, may end up creating a selffulfilling
prophecy—research students (and perhaps academics) who
join the department might do so partly because of the emphasis
placed on particular methodologies and research areas. MLS students
need exposure to a wide range of methods given the wide range of
professional contexts in which future research may be carried out or
encountered. On the other hand, no methods course could cover every
method ever devised. When deciding which methods need to be
covered, and to what depth, course developers may wish to consider
which methods are the most commonly used in LIS research—or at
least, those methods most commonly used in quality LIS research. In
MLS programs, it may be appropriate to define LIS research primarily
in terms of the research carried out by or for practitioners. In doctoral
programs, research may be appropriately defined in terms of that
carried out within the academy.
1.1. Problem statement
LIS researchers and research students need to be familiar with the
strategies and techniques they are likely to encounter, and possibly
select, as they go about their research. The problem is that LIS is a very
broad discipline, or meta-discipline, that uses a wide variety of continuously
evolving strategies and techniques. Researchers with a finite
amount of time cannot fully master all the strategies and techniques
on offer. They must therefore be selective and focused. To this end, it is
important for them, and for those designing methods courses in LIS
doctoral programs, to know which research strategies and techniques
are being applied in current high-profile LIS academic research. Given
that LIS methods courses generally focus on empirical research, this
study does not cover non-empirical strategies, such as conceptual and
mathematical methods. High-profile research was defined in terms of
that reported in top-rated research journals.
Library & Information Science Research 30 (2008) 108-114
⁎ Corresponding author.
E-mail addresses: phider@csu.edu.au (P. Hider), rpymm@csu.edu.au (B. Pymm).
a r t i c l e i n f o
Article history:
Available online 27 May 2008
0740-8188/$ – see front matter © 2008 Elsevier Inc. All rights reserved.
doi:10.1016/j.lisr.2007.11.007
Contents lists available at ScienceDirect
Library & Information Science Research
The two primary research questions were:
1. Which strategies and techniques are most used in current, highprofile
LIS research?
2. What are the trends in the use of LIS research methods?
The second question was answered by comparison with earlier
studies based on 1975 and 1985 literature.
2. Literature review
Several content analyses of LIS research literature have been conducted
previously, though most of them are somewhat dated. Bernhard
(1993) provided a helpful summary of earlier studies. She combined
earlier results with a study of the coverage of methods in various
textbooks and reference works to produce a list of thirteen methods
considered the most important in the discipline. Her analysis shows a
good deal of convergence among the earlier studies in terms of both
classification and prevalence of methods. Most of the studies are based
on journal literature, though some are based on secondary sources or
dissertations. In addition, some compare different periods (generally
defined as a year). Most examine other aspects as well as methodology,
particularly subject.
Järvelin and Vakkari (1990) criticized some of the studies cited by
Bernhard for a lack of refinement in their analysis of methodology and
method. By contrast, their own study broke methodology and method
down into four variables: type of investigation, research strategy, data
collection method, and type of analysis. The type of investigation
might be regarded as the general methodological approach—methodology
types include empirical, descriptive, comparative, and conceptual.
Although Järvelin and Vakkari (1990) also had a no research
category, they defined research very broadly. For example, elsewhere,
research papers might be distinguished from conceptual, descriptive,
or professional papers. Most of the types of investigation correspond
to one or more research strategies. The empirical type was the most
subdivided, accounting for twelve strategies. In current methods
language, these strategies would often be referred to as specific
methodologies (Crotty, 1998). Järvelin and Vakkari (1990) went on to
classify their empirical research strategies according to two other
variables: first, in terms of data collection method (which elsewhere
might be referred to as technique) and second, in terms of data
analysis (qualitative or quantitative). These two additional variables
appear to be a useful refinement because particular strategies do not
necessarily entail particular collection or analytical techniques.
Bernhard (1993) included Järvelin and Vakkari's (1990) study and
drew from their strategies for her list of research methods:
Content analysis
Information systems design
Survey
Bibliometrics
Comparative studies
Case studies
Ethnographic research—naturalistic enquiry
Evaluative studies
Experimental research
Historical research
Theory development
Delphi method
Operations research—systems analysis
Bernhard also offers definitions for each method, something
surprisingly lacking in the reporting of most of the content analyses.
Much of Bernhard's list is covered in the strategy classification devised
by Järvelin and Vakkari (1990), including information systems design
(which they placed outside of the empirical research strategies).
Ethnographic research and Delphi method were presumably covered
by Järvelin and Vakkari's qualitative method and survey, while theory
development was included in their conceptual research strategies
(verbal argumentation and conceptual analysis). On the other hand,
Järvelin and Vakkari (1990) did not list comparative studies as a strategy,
onlyas a type of investigation. They did not list operations research at all.
Järvelin and Vakkari's (1990) analysis is one of the more recent and
sophisticated, even though it is based on articles published in 1985. They
used 37 core LIS journals, identified by such criteria as widespread
distributions and international editorial boards. They argued that their
exclusive use of journal articles would not significantly bias results,
particularly because much of the research reported in monographic and
other non-journal literature is alsowritten up in journals. Theyexamined
a total of 449 research articles (according to their broad definition of
research). They then classified the articles according to seven variables,
including the four described above. For each of the variables, Järvelin and
Vakkari designed categories to be mutually exclusive. Presumably they
determined the primary strategy and technique in cases where mixed
strategies and techniques were deployed and reported.
Kumpulainen (1991) followed up on Järvelin and Vakkari's (1990)
study. She used the same classification scheme to investigate the
research areas and methods of 359 research articles in LIS journal
literature from 1975. Her list of 30 journals was based in part on
Järvelin and Vakkari's list, though she revised it to reflect core journals
from a decade earlier.
There has been little content analysis performed on more recent
research literature with respect to research methods. Julien (1996)
looked at methods reported in the information needs and uses literature
from 1990–1994. Clyde (2004) examined school librarianship
literature from 1991–2000. Blake's (1994) study of LIS dissertation
abstracts from 1975–1989 revealed some differences in the distributions
of methods used by library science and information science
research students. Much more experimentation was performed by the
latter, though the survey was still the most popular across LIS. Blake
also detected greater methodological variation in information science,
the demise of the historical method, and the general reduction in
numbers of library science dissertations. Outside of content analysis,
Powell (1999) identified some emerging trends in both qualitative and
quantitative research. There has also been a fair amount of discussion
about the place of research in the profession, culminating in the rise of
evidence-based librarianship (Crumley & Koufogiannakis, 2002).
Meanwhile, both Park (2004) and Morris (2006) have conducted a
different kind of content analysis—analyzing the documented curricula
themselves. Park examined methods subjects taught in the US and
Korea, while Morris looked at UK courses. They both found wide
variations across departments. Some emphasized qualitative methods;
others emphasized quantitative methods. There were also marked
differences in course length and subject depth. This appears to reflect
the wide range of methods employed across the discipline, as Eldredge
(2004) described in his inventory of LIS methods.
3. Research design
3.1. Construction of classification scheme
This paper describes a content analysis of contemporary LIS research
literature, which aimed to identify the most common strategies
and techniques employed by LIS researchers carrying out high-profile
empirical research. The study utilized the relevant parts of Järvelin and
Vakkari (1990) classification scheme. Their taxonomies were relatively
refined. They were used by subsequent researchers and established
with reference to empirical data, which allowed comparison with
previous years—1975 (Kumpulainen, 1991) and 1985 (Järvelin &
Vakkari 1990).
This study's strategy classification included Järvelin and Vakkari's
(1990) eleven empirical research strategies, even though the secondary
109P. Hider, B. Pymm / Library & Information Science Research 30 (2008) 108–114
analysis strategy overlapped with the other ten (and wasnot particularly
useful). However, recent literature labels many of these strategies as
methodologies rather than methods. Therefore, this study revised the
terminology to avoid the term method. Also, one extra category was
included: mixed strategies. This study excluded all of Järvelin and
Vakkari's (1990) non-empirical strategies, including their system and
software analysis and design and operations research strategies. All
other strategies not specifically covered by the classification were
classed under other strategy, unless they were qualitative—in which case
they were classed under qualitative strategy. The classification includes
categories which might be considered research designs or methods, as
well as those that might be considered methodologies. However, the
scheme was retained in order to facilitate comparisonwith the results of
the earlier studies. All eleven categories are listed below.
Strategy
Historical research
Survey
Qualitative strategy
Evaluation
Case or action research
Content or protocol analysis
Citation analysis
Other bibliometric analysis
Secondary analysis
Experiment
Other strategy
Mixed strategies
Some of the strategies in the classification have been variously
defined and might be considered to overlap with other categories. To
assist the analyst, the relevant definitions proposed by Bernhard
(1993) were adopted. For the purposes of this study, action research is
subsumed under Bernhard's case study definition. Protocol analysis is
subsumed under her content analysis definition.
Corresponding definitions did not exist, however, for certain terms.
Citation analysis and other bibliometric analysis were treated together by
Bernhard, and secondary analysis and qualitative strategy were very
generic categories. This study defined citation analysis in terms of
Bernhard's definition for bibliometrics, where the principal unit(s) of
measurement related to citations. Other bibliometric analysis used the
same definition, but the principalunit(s)ofmeasurement did notrelateto
citations. Where there was a mix of citation and non-citation bibliometrics,
citation analysis took precedence. Secondary analysis was defined
narrowly as any form of method that was used to answer a research
question of a previous study using the same data (i.e., pre-existing data).
Several of the other categories might be considered qualitative
methods—at least sometimes. Therefore, the definition of qualitative
strategy was any strategy based on the qualitative analysis of data that
did not fall into one of the other categories (excluding the other
strategy category). This strategy corresponded to the qualitative type
of analysis, though it did not account for the only instances of it.
This study used two other parts of Järvelin and Vakkari's (1990)
scheme to shed further light on the strategies and techniques
employed in contemporary LIS research: the array of data collection
techniques (which are sometimes termed methods in the literature)
and the qualitative and quantitative categorizations under type of
analysis. Essentially, the same nine categories were used to classify
data collection. A further type of analysis was added, namely, that
which was both qualitative and quantitative. The two arrays are listed
below.
Data collection technique
Questionnaire, interview
Observation
Thinking aloud
Content analysis
Citation analysis
Historical source analysis
Use of data collected earlier
Other technique
More than one technique
Type of analysis
Qualitative
Quantitative
Both
The categories in each of the three classification facets were treated
as mutually exclusive for the purposes of this analysis—an article was
classed in only one category. Where there was clearly one primary
strategy or technique, the article was classed in the corresponding
category; where there was more than one primary strategy or
technique, it was classed with the mixed strategies or more than one
technique category. Similarly, if major parts of the analysis were
quantitative but other major parts were qualitative, the both category
was used.
3.2. Selection of literature
Not all high-quality research is reported in high-profile journals,
nor do all the articles in high-profile journals necessarily represent
high-quality research. Nevertheless, the study assumes that there is a
great deal of overlap. It is worth identifying strategies and techniques
used in high-profile research, even if it is not all high quality, as such
research (or the reporting of such research) is likely to influence future
research directions.
Following other studies, such as those of Järvelin and Vakkari (1990)
and Kumpulainen (1991), this study examined only journal articles for
practical reasons. This study assumed that the top journals in the LIS
field were at least as influential as the top monographs, conference
papers, reports, and dissertations. It also assumed that the journals are
reasonably representative of current research (and probably more so
than monographic literature), and that the journals selected represent
a broad cross-section of LIS and related research. Järvelin and Vakkari
(1990) and Kumpulainen (1991) selected core LIS journals in terms of
importance and relevance. This study emphasized the criterion of
importance in the senses of impact and profile. As the basis for
selection, it used the ranked list of journals by impact factor in the 2005
Institute of Scientific Information (ISI) journal citation report for the
field (Institute of Scientific Information, n.d.). All articles in 2005 issues
(according to their chronological designations) of 20 of the top 30
journals were examined. (There was one exception where access to the
last 2005 issue of a journal could not be gained, and so the last issue
from 2004 was used instead). Nine of the other ten journals were
unable to be analyzed due to lack of access to the full text, while another
journal (Annual Review of Information Science & Technology) contained
only review articles. The 20 journals examined are listed in Table 1.
For the analysis, the authors discounted articles that were less
substantial, such as editorials, book reviews, obituaries, and brief
communications. The authors also excluded historical reprints from
the analysis. Of the remaining articles, those deemed not to report
new empirical research were assigned to the separate category no
empirical research. This category included discussion papers and
essays, literature reviews, theoretical papers, and articles featuring
conceptual analyses, information systems design, simulations, and so
forth. All other articles were assigned one category from each of the
three facets in the classification scheme outlined above.
It is assumed that these articles, by virtue of appearing in these top
journals enjoy a relatively high profile. Although it has been argued
that citation impact factors favor the harder sciences, whose distribution
of research strategies and techniques is different from that of
110 P. Hider, B. Pymm / Library & Information Science Research 30 (2008) 108-114
the softer sciences, they have been employed extensively across the
broad spectrum of disciplines as indicators of quality. Given this
amount of use, they are perhaps even better approximations of
reputation. In any case, of the 20 journals examined, at least 6 hail
from the softer, librarianship side of the discipline. Moreover, the list
overlaps significantly (despite its being for 1985) with Järvelin and
Vakkari's (1990) list of core journals, which was compiled on a more
qualitative basis.
Another way in which the ISI list might be somewhat biased is in
its greater coverage of publications from English-speaking countries.
Although this may not be viewed as a major issue for LIS departments
in English-speaking countries, it is worth bearing in mind.
3.3. Reliability testing
Parallel coding was conducted by the two authors for all the 2005
issues of four of the journals. A formative approach was adopted,
whereby the authors discussed mismatches and assessed their rate of
agreement as they progressed through the initial issues. In addition to
percentage agreements of well over 80%, Cohen's kappa coefficient
was used to assess the level of inter-coder agreement for the third and
fourth journals, a total of 52 articles. The coefficients were 0.895 for
the strategy variable, 0.788 for the technique variable, and 0.810 for
the type of analysis variable. All three scores were considered
sufficiently high for coding to proceed separately, particularly given
the conservative nature of the kappa coefficient. By this time, it had
also become apparent that further improvement in consistency would
be curbed by some fuzziness in the strategy taxonomy, which gave rise
to instances in which both authors considered that a case could be
made for more than one category. This matter is discussed further in
the last section of this paper. The remaining coding of the other
journals was shared equally by the two authors. Both authors found it
necessary to access the full text of all articles in order to identify the
various facets of each paper. All had abstracts, but in most cases they
were insufficient to enable accurate coding.
4. Results
The overall counts for the three facets in the classification scheme
are shown in Tables 2–4. The front-runner was the survey, with almost
a third of the total. Experimentation was a clear second. The other
strategies scored between 4 and 9%, apart from historical research and
secondary analysis, which both had very low numbers. The leading
technique was questionnaire/interview, with about a third of the total.
This was not surprising, given the popularity of the survey. The only
other specific technique with more than 10% was content analysis. The
use of pre-existing data, other techniques, and multiple techniques all
scored between 14 and 17%. The relatively modest count for multiple
techniques means that any difference in distribution among the
individual techniques hidden within the coding for the multiple
techniques category would likely have little bearing on the overall
scores for particular techniques; the same may be said for mixed
strategies. For type of analysis, about two-thirds of the research was
quantitative, a long way ahead of qualitative research and research
using both types of analysis.
The proportion of articles reporting no empirical research was
32.0% (267/834). This high figure is due to many discussion papers,
some theoretical papers, and some papers in the information sciences
that featured modelling and simulation exercises (particularly those
overlapping with computer and management sciences).
The journals examined represent a wide range of fields spanning
the whole spectrum of LIS, as well as some journals that overlapped
with other disciplines (such as computer science and management
science). Therefore, the authors decided to compare the results from
journals that focus primarily on librarianship with the results from
those that do not. The librarianship journals were defined as those
focused primarily on topics pertaining to libraries (or a particular type
of library) and included College & Research Libraries, The Journal of the
Medical Library Association, Library & Information Science Research,
Library Quarterly, The Journal of Academic Librarianship, and Library
Resources & Technical Services. The comparison is shown in Tables 5–7.
Interestingly, the proportion of articles reporting no empirical
research in the librarianship journals was 33.0% (69/209), compared
with 31.7% (198/625) in the other journals. There was no significant
difference, despite an expectation that librarianship journals might
contain more discussion papers and essays.
Finally, the overall results for 2005 were compared with those
for 1985 and 1975 produced by Järvelin and Vakkari (1990) and
Table 2
Strategies used in 2005
Strategy n %
Historical research 7 1.2
Survey 173 30.5
Qualitative strategy (other) 31 5.5
Evaluation 23 4.1
Case or action research 48 8.5
Content or protocol analysis 27 4.8
Citation analysis 26 4.6
Other bibliometric analysis 32 5.6
Secondary analysis 4 0.7
Experiment 118 20.8
Other strategy 49 8.6
Mixed strategies 29 5.1
Total 567 100.0
Table 3
Data collection techniques in 2005
Data collection technique n %
Questionnaire, interview 184 32.5
Observation 21 3.7
Thinking aloud 1 0.2
Content analysis 59 10.4
Citation analysis 26 4.6
Historical source analysis 7 1.2
Use of data collected earlier 95 16.8
Other technique 81 14.3
More than one technique 93 16.4
Total 567 100.0
Table 1
Journals examined
Journal Impact
factor
Rank
Management Information Systems Quarterly 4.978 1
Journal of the American Medical Information Association 4.339 2
Information Systems Research 2.054 4
Journal of the American Society for Information
Science and Technology
1.583 6
International Journal of Geographical Information Science 1.562 7
Journal of Management Information Systems 1.406 10
College and Research Libraries 1.245 11
Journal of the Medical Library Association 1.225 12
Information Processing and Management 1.192 13
Information Society 1.018 14
Journal of Documentation 0.983 15
Library and Information Science Research 0.957 16
Journal of Information Science 0.747 19
Information Research 0.701 20
Library Quarterly 0.688 21
Information Systems Journal 0.559 23
Journal of Academic Librarianship 0.559 23
Library Resources and Technical Services 0.512 27
International Journal of Information Management 0.479 29
Research Evaluation 0.474 30
111P. Hider, B. Pymm / Library & Information Science Research 30 (2008) 108–114
Kumpulainen (1991). These are shown in Tables 8–10 . The mixed
strategies category was discounted from the 2005 analysis because it
was not employed in the earlier analyses. Secondary analysis was also
discounted because its result was not shown in the earlier figures.
Similarly, for type of analysis, the both category was omitted from the
2005 figures.
5. Discussion
5.1. Comparison of distributions
With respect to the distributions of strategies reported in the
librarianship and non-librarianship journals, the degree of correlation
was no more than fair. This indicated some significant differences in
the research approaches adopted by librarianship researchers and
other LIS researchers. The proportions differed significantly for several
strategies. The survey was even more prevalent in the librarianship
journals, accounting for more than half of the research. Conversely,
there were no experiments recorded in the librarianship journals—
dramatically different from the 27.6% in the other journals. Historical
research and content analysis occurred a little more often in the
librarianship journals, whereas non-citation forms of bibliometrics
occurred less.
For the techniques, the degree of correlation between librarianship
and non-librarianship was fair, but again, it could have been higher.
Given the prevalence of surveys in the librarianship journals, it was
not surprising that questionnaires and interviews also featured very
prominently. Another notable statistic was that non-librarianship
journals had a far greater proportion of research using data collected
previously. This was partly due to the use of test collections in the
information retrieval field, barely covered by the librarianship
journals. There were several other often-encountered techniques
that could not be specified in the analysis using Järvelin and Vakkari's
(1990) categories. These included task analysis (the often-automated
measurement of task accomplishments) and dataset construction for
experimentation. These techniques were more often adopted by
researchers publishing in the non-librarianship journals.
Quantitative research still accounted for more than half of the
librarianship journal articles, though the percentage was higher in
non-librarianship journals. The proportions of qualitative research
were similar (around 20%). The lesser amount of quantitative analysis
in librarianship research was compensated instead by more research
combining quantitative and qualitative analysis.
Some trends can be observed when comparing the 1975, 1985, and
2005 distributions. There is an extreme trend downwards for historical
research, as well as marked trends against evaluation and survey.
Considering the findings for the librarianship journals, the decreases
in historical research and survey may well be linked, at least in part, to
a decrease in the coverage of traditional librarianship concerns in the
high-profile journal literature. Conversely, there was a very marked
increase in experimentation, no doubt due in part to the abundance of
articles in the areas of information retrieval and computer science.
There are also significant increases, although from lower bases, for
qualitative strategies (outside of history) and content analysis.
Somewhat surprisingly, the use of questionnaires and interviews
appears to have increased even though the survey has declined. This
can be explained, however, by the rise of qualitative research, which is
often conducted through interviews. The use of historical source
material has declined in correspondence to the decline in the historical
research.
Interestingly, there does not appear to have been any great overall
shift in the proportions of quantitative and qualitative research. An
increase in the use of more sophisticated qualitative research methods
has been balanced by the decrease in historical research and by an
increase in experimentation.
It is also worth noting that the percentage of articles not reporting
empirical research was 55.9 in the 1985 analysis, compared with 32.0
in 2005. It is quite possible that the earlier study covered articles that
were not examined in the later study. However, the large difference
suggests that certain types of article, such as discussion and
descriptive articles, may be less likely to find their way into the top
LIS journals today than in previous times.
5.2. Revision of classification scheme
Other strategies ranked third in the 2005 results, behind only
surveys and experiments. Of the non-specified strategies, there were
significant numbers of similar type. This study proposes that a revised
classification scheme includes more specific strategies and clearer
definitions so as to reduce the number of others.
Table 4
Types of analysis in 2005
Type of analysis n %
Qualitative 112 19.8
Quantitative 367 64.7
Both 88 15.5
Total 567 100.0
Table 5
Librarianship and non-librarianship strategies
Strategy Librarianship Non-librarianship
n % n %
Historical research 5 3.6 2 0.5
Survey 72 51.4 101 23.7
Qualitative strategy (other) 9 6.4 22 5.2
Evaluation 4 2.9 19 4.4
Case or action research 11 7.9 37 8.7
Content or protocol analysis 12 8.6 15 3.5
Citation analysis 8 5.7 18 4.2
Other bibliometric analysis 5 3.6 27 6.3
Secondary analysis 0 0.0 4 0.9
Experiment 0 0.0 118 27.6
Other strategy 7 5.0 42 9.8
Mixed strategies 7 5.0 22 5.2
Total 140 100.0 427 100.0
Table 6
Librarianship and non-librarianship data-collection techniques
Data collection technique Librarianship Non-librarianship
n % n %
Questionnaire, interview 60 42.9 124 29.0
Observation 4 2.9 17 4.0
Thinking aloud 0 0.0 1 0.2
Content analysis 21 15.0 38 8.9
Citation analysis 8 5.7 18 4.2
Historical source analysis 4 2.9 3 0.7
Use of data collected earlier 6 4.3 89 20.8
Other technique 11 7.9 70 16.4
More than one technique 26 18.6 67 15.7
Total 140 100.0 427 100.0
Table 7
Librarianship and non-librarianship analyses
Type of analysis Librarianship Non-librarianship
n % n %
Qualitative 32 22.9 80 18.7
Quantitative 72 51.4 295 69.1
Both 36 25.7 52 12.2
Total 140 100.0 427 100.0
112 P. Hider, B. Pymm / Library & Information Science Research 30 (2008) 108-114
As mentioned earlier, the overlap between strategies in the scheme
limits the reliability of its use. Notwithstanding the definitions that
were adopted for this analysis, experiments and surveys can also be
evaluations, and so on, without such cases necessarily representing a
mixed methods approach. Rather, an approach may simply reflect
aspects of two or even three strategies. In addition to constructing
tighter definitions (which is necessary), a way forward might be to
establish an order of priority. However, it is difficult to justify any
particular order. Instead, the fuzziness of the strategy categories needs
to be recognized—distributions resulting from content analyses based
on them can only be approximate representations of researchers'
approaches. Similarly, it should be recognized that researchers often
approach a problem or question in a multi-dimensional way.
A major problem with the strategy classification is the presence of
categories that really represent techniques, as reflected in their
duplication in the technique taxonomy. It is thus recommended that
content analysis, protocol analysis (which need not be merged with
content analysis), citation analysis, and other bibliometric analysis
techniques are dropped from the list of strategies. This would also
reduce the overlap problem.
The authors also do not see any need to amalgamate case study and
action research; instances of case study were often quite distinct from
instances of action research. It is recommended that action research
forms a separate strategy category.
Another problem with the strategy classification was using qualitative
as a catch-all category. There was significantly more qualitative
analysis being carried out in 2005 than in 1985, the year for which the
scheme was originally constructed. There is, in any case, no point in
using a category which is defined solely on the basis of the analysis
facet. Instead, it is proposed that the category be dropped and
replaced by ethnography. This allows for other qualitative approaches,
to be classed as other strategy if they are not covered by the other
specific strategies in the array. Other quantitative approaches, as well
as ones that are both quantitative and qualitative, would also be
classified as other strategy. Ethnography would usually include studies
based on participant observation, journal entries, unstructured interviews,
and so on. Case study research, on the other hand, would focus
on particular cases; it may or may not employ a sophisticated
methodology such as grounded theory.
Finally, there seems little reason to retain secondary analysis—it is
not in itself a research strategy and rarely occurs. In summary, it is
recommended that the strategy facet be revised as below.
Strategy
Historical research
Ethnography
Case study
Action research
Survey
Evaluation
Experimentation
Other strategy
Mixed strategies
This array coincides to varying degrees with those methods
covered in LIS textbooks. For example, it coincides with six out of
Williamson's eight methods (Williamson, 2002), and four out of
Powell and Connaway's (2004) nine methods (pp. 59–68). It shares
only 2 or 3 of 22 methods in Eldredge's recent inventory (Eldredge,
2004). However, Eldredge includes some very broad methods, such as
analysis and summing up, as well as some much narrower techniques.
Powell and Connaway also included techniques that have been
deleted from the new taxonomy recommended here.
Just as there were quite a few other strategies in our findings, so too
were there a significant number of other techniques. Techniques that
were not specifically covered but were sometimes reported included
transaction log analysis, focus groups, task analysis, journal entries,
and bibliometric techniques outside of citation analysis. Many other
instances of other technique involved the construction of data sets for
information retrieval experiments. Although citation analyses could
largely be distinguished from other forms of bibliometric analysis,
there seems little practical reason why they need to be categorized
separately. Researchers recommend that bibliometric analysis be made
a unifying category that embraces newer forms of webliometrics, such
as link analysis.
There was less of a problem with overlap between technique categories
than between strategies. However, certain categories encompassed
fairly diverse techniques. Observation was used for certain
ethnographic studies but also for usability inspections and the like.
Researchers propose adding the category inspection instead. This
category would often be undertaken by experts or users and would
often involve the use of checklists, etc. Use of data collected earlier often
meant the use of test collections for information retrieval experiments,
but a range of other data was used. However, given the nature of the
technique—using previously collected and processed data—it does not
seem necessary to break this category down any further.
In summary, it is recommended that the technique facet be revised
as below.
Data collection technique
Historical source analysis
Questionnaire, interview
Focus groups
Journal entries
Observation
Inspection
Protocol analysis
Content analysis
Table 8
Trends in LIS research strategies (%)
Strategy 1975 1985 2005
Historical research 24.7 19.1 1.2
Survey 40.1 41.0 30.5
Qualitative strategy (other) 0.0 2.9 5.5
Evaluation 17.0 10.0 4.1
Case or action research 3.8 6.8 8.5
Content or protocol analysis 0.0 2.0 4.8
Citation analysis 3.8 5.9 4.6
Other bibliometric analysis 2.2 1.6 5.6
Experiment 7.7 2.9 20.8
Other strategy 0.5 8.1 8.6
Table 9
Trends in LIS research techniques (%)
Data collection technique 1975 1985 2005
Questionnaire, interview 21 26 32.5
Observation 3 2 3.7
Thinking aloud 0 0 0.2
Content analysis 4 4 10.4
Citation analysis 4 7 4.6
Historical source analysis 25 19 1.2
Use of data collected earlier 12 9 16.8
Other technique 18 26 14.3
More than one technique 13 4 16.4
Table 10
Trends in LIS research analysis (%)
Type of analysis 1975 1985 2005
Qualitative 39.5 17.1 23.4
Quantitative 60.5 82.9 76.6
113P. Hider, B. Pymm / Library & Information Science Research 30 (2008) 108–114
Bibliometric analysis
Transaction log analysis
Task analysis
Dataset construction
Use of data collected earlier
Other technique
More than one technique
There appears to be no particular need to revise the type of analysis
facet beyond the revision made for this study (i.e., adding a both
category). The authors' coding rarely disagreed for this facet.
5.3. Future research
There are various ways to replicate this content analysis. It would, of
course, be interesting to see what the distributions of research strategies
and techniques look like in 2015, given the changes exposed in this
study. It would also be interesting to see what they look like for lowerprofile
research and that featured in the professional literature. The
latter is the kind of research more likely to be undertaken
by practitioners, towards whom research methods courses in MLS
programs are probably best geared. Given that one of the chief purposes
of this research was to provide an indication of what courses students
need to cover for LIS research, it would also be worthwhile to perform a
parallelcontent analysis on the currentcoursesyllabi to gauge the extent
to which they are in step with current research practice.
6. Conclusion
The results of the content analysis suggest that LIS researchers who
publish in the top journals today mostly rely on a couple of strategies:
survey and experimentation. However, researchers also used a wide
range of other strategies in fairly significant numbers, demonstrating
the broad church that the discipline (or group of disciplines) embodies.
Qualitative approaches including case studies and ethnography are
now well established, but quantitative techniques such as bibliometrics
still figure as valuable tools of investigation. Indeed, the largely
quantitative technique of transaction log analysis has grown rapidly to
become a major instrument for those researching the performance of
very large databases and related information retrieval issues.
It is thus recommended that coverage is given to at least the
following strategies in LIS doctoral programs: survey, experimentation,
case study, ethnography, and evaluation. This would ensure that
certain techniques closely associated with certain strategies are
covered, including questionnaires, interviews, content analysis, bibliometrics,
and transaction log analysis.
Comparison with earlier studies indicates a discipline (or set of
disciplines) that continues to evolve and utilize new approaches and
techniques as they emerge. The emphasis on empirical research
appears stronger than ever as LIS seeks to cement its position in the
academic community. The diversity and sophistication of strategies
and techniques currently applied by LIS researchers bodes well for the
future, with a healthy mix of quantitative and qualitative approaches
reflecting the many kinds of research question that LIS addresses. LIS
researchers need to be keenly aware of this methodological diversity
in order to build on it.
References
Bernhard, P. (1993). À la recherche des… methods de recherche utilisées en sciences de
l'information. Canadian Journal of Information & Library Sciences, 18, 1−35.
Blake, V. L. P. (1994). Since Shaughnessy: Research methods in library and information
science dissertations, 1975–1989. Collection Management, 19(1/2), 1−42.
Crumley, E., & Koufogiannakis, D. (2002). Developing evidence-based librarianship:
Practical steps for implementation. Health Information & Libraries Journal, 19(2),
61−70.
Clyde, L. A. (2004). Research in school librarianship 1991–2000: Australia in an international
setting. Australian Library Journal, 53(2), 181−199.
Crotty, M. (1998). The foundations of social research. St. Leonards, Australia: Allen &
Unwin.
Eldredge, J. D. (2004). Inventory of research methods for librarianship and informatics.
Journal of the Medical Library Association, 92(1), 83−90.
Institute of Scientific Information. (n.d.). Journal citation reports. Retrieved June 1, 2006,
from ISI Web of Knowledge.
Järvelin, K., & Vakkari, P. (1990). Content analysis of research articles in library and
information science. Library & Information Science Research, 12, 395−422.
Julien, H. (1996). A content analysis of the recent information needs and uses literature.
Library & Information Science Research, 18, 53−65.
Kumpulainen, S. (1991). Library and information science research in 1975: Content
analysis of the journal articles. Libri: International Journal of Libraries & Information
Services, 41(4), 59−76.
Morris, A. (2006). Provision of research methods teaching in UK LIS departments. New
Library World, 107(3/4), 116−126.
Park, S. (2003). Research methods as a core competency. Journal of Education for Library
& Information Science, 44(1), 17−25.
Park, S. (2004). The study of research methods in LIS education: Issues in Korean and U.
S. universities. Library & Information Science Research, 26, 501−510.
Powell, R. R. (1999). Recent trends in research: A methodological essay. Library &
Information Science Research, 21, 91−119.
Powell, R. R., & Connaway, L. S. (2004). Basic research methods for librarians (4th ed.).
Westport, CT: Libraries Unlimited.
Williamson, K. (2002). Research methods for students, academics and professionals:
Information management and systems (2nd ed.). Wagga Wagga, N.S.W., Australia:
Centre for Information Studies, Charles Sturt University.
114 P. Hider, B. Pymm / Library & Information Science Research 30 (2008) 108-114