Measuring information systems success:
models, dimensions, measures,
and interrelationships
Stacie Petter1
,
William DeLone2
and
Ephraim McLean3
1
Department of Information Systems and
Quantitative Analysis, University of Nebraska at
Omaha, 6001 Dodge Street, PKI 173B, Omaha,
NE 68182, U.S.A.; 2
Department of Information
Technology, American University, 4400
Massachusetts Ave. NW, Washington DC
20016, U.S.A.; 3
Department of Computer
Information Systems, Robinson College of
Business, Georgia State University, Atlanta,
GA 30303, U.S.A.
Correspondence: Ephraim McLean,
Department of Computer Information
Systems, Robinson College of Business,
Georgia State University, Atlanta,
GA 30303, USA.
Tel: 1 404 413 7448;
E-mail: emclean@gsu.edu
Received: 30 December 2006
Revised: 12 July 2007
2nd Revision: 16 December 2007
3rd Revision: 29 April 2008
Accepted: 15 May 2008
Abstract
Since DeLone and McLean (D&M) developed their model of IS success, there
has been much research on the topic of success as well as extensions and tests
of their model. Using the technique of a qualitative literature review, this
research reviews 180 papers found in the academic literature for the period
1992–2007 dealing with some aspect of IS success. Using the six dimensions of
the D&M model – system quality, information quality, service quality, use, user
satisfaction, and net benefits – 90 empirical studies were examined and the
results summarized. Measures for the six success constructs are described and
15 pairwise associations between the success constructs are analyzed. This
work builds on the prior research related to IS success by summarizing the
measures applied to the evaluation of IS success and by examining the
relationships that comprise the D&M IS success model in both individual and
organizational contexts.
European Journal of Information Systems (2008) 17, 236–263. doi:10.1057/ejis.2008.15
Keywords: information systems success; organizational and individual use of IS; IS
effectiveness; IT performance; measurement; literature review
Introduction
In 2008, organizations continue to increase spending on information
technology (IT) and their budgets continue to rise, even in the face of
potential economic downturns (Kanaracus, 2008). However, fears about
economic conditions and increasing competition create pressures to cut
costs, which require organizations to measure and examine the benefits
and costs of technology. Naturally, organizations are interested in knowing
the return on these investments. The impacts of IT are often indirect and
influenced by human, organizational, and environmental factors; therefore,
measurement of information systems (IS) success is both complex and
illusive.
IS are developed using IT to aid an individual in performing a task. Given
the relatively short life-span of the IS field, it is quite remarkable the
number and variety of applications and systems that have been deployed.
There are IS that range from hedonic, developed for pleasure and
enjoyment, to utilitarian, developed to improve individual and organizational
performance (van der Heijden, 2004). Organizations focus on
developing, using, and evaluating utilitarian IS. There is a plethora of
utilitarian IS used in organizations, such as decision support systems,
computer-mediated communications, e-commerce, knowledge management
systems, as well as many others.
European Journal of Information Systems (2008) 17, 236–263
& 2008 Operational Research Society Ltd. All rights reserved 0960-085X/08
www.palgrave-journals.com/ejis
To measure the success of these various IS, organizations
are moving beyond traditional financial measures,
such as return on investment (Rubin, 2004). In an effort
to better understand the tangible and intangible benefits
of their IS, organizations have turned to methods such as
balanced scorecards (Kaplan & Norton, 1996) and
benchmarking (Seddon et al., 2002). Researchers have
created models for success (DeLone & McLean, 1992;
Ballantine et al., 1996; Seddon, 1997), emphasizing the
need for better and more consistent success metrics.
As a field, we have made substantial strides towards
understanding the nature of IS success. For example, the
widely cited DeLone and McLean (D&M) model of IS
success (1992) was updated a decade later based on a
review of the empirical and conceptual literature on IS
success that was published during this period (DeLone &
McLean, 2003). Furthermore, some researchers have
synthesized the literature by examining one or more of
the relationships in the D&M IS success model using the
quantitative technique of meta-analysis (Mahmood et al.,
2001; Bokhari, 2005; Sabherwal et al., 2006) to develop a
better understanding of success. Others have started to
develop standardized measures that can be used to
evaluate the various dimensions of IS success as specified
by D&M (e.g., Sedera et al., 2004). This past research has
helped the field better understand IS success, but more
needs to be done. Therefore, this paper examines the
research related to IS success to determine what is known
and what still needs to be learned for utilitarian IS in an
organizational context.
This qualitative literature review identified three
objectives for improving the current understanding of
the literature in this domain. First, the D&M model was
examined in two different contexts: the individual level
of analysis and the organizational level of analysis in
order to identify if the unit of analysis under study is a
boundary condition for measuring success. Second, unlike
other literature reviews or meta-analyses that have
only reviewed some of the relationships in the original
D&M model, this review investigated all relationships in
the updated IS success model (DeLone & McLean, 2003).
Finally, the specific measures used by researchers for each
of the constructs that comprise the D&M model were
examined. In both the original and updated models,
D&M strongly advocated the need for consistent and
appropriate measures for IS success. This review seeks to
determine whether researchers have heeded this call.
The next section explores various competing models of
IS success and explains why the D&M model was chosen
as the organizing framework for this literature review. A
description of the methods used to obtain and classify the
research, as well as a more detailed description of the
literature review, is also provided. The results of this
literature review are then presented, organized by the
constructs contained in the D&M IS success model as well
as the relationships between the constructs. The final
section outlines the implications of this body of research
for both practitioners and researchers interested in
assessing information system success.
Background
The development of a model of IS success
Researchers have derived a number of models to explain
what makes some IS ‘successful.’ Davis’s (1989) Technology
Acceptance Model (TAM) used the Theory of
Reasoned Action and Theory of Planned Behavior (Fishbein
& Ajzen, 1975) to explain why some IS are more
readily accepted by users than others. Acceptance,
however, is not equivalent to success, although acceptance
of an information system is a necessary precondition
to success.
Early attempts to define information system success
were ill-defined due to the complex, interdependent, and
multi-dimensional nature of IS success. To address this
problem, DeLone & McLean (1992) performed a review of
the research published during the period 1981–1987, and
created a taxonomy of IS success based upon this review.
In their 1992 paper, they identified six variables or
components of IS success: system quality, information
quality, use, user satisfaction, individual impact, and
organizational impact. However, these six variables are
not independent success measures, but are interdependent
variables. Figure 1 shows this original IS success model
(DeLone & McLean, 1992).
Shortly after the publication of the D&M success
model, IS researchers began proposing modifications to
this model. Accepting the authors’ call for ‘further
System
Quality
Information
Quality
Use
User
Satisfaction
Individual
Impact
Organizational
Impact
Figure 1 DeLone and McLean IS success model (1992).
Measuring information systems success Stacie Petter et al 237
European Journal of Information Systems
development and validation,’ Seddon & Kiew (1996)
studied a portion of the IS success model (i.e., system
quality, information quality, use, and user satisfaction).
In their evaluation, they modified the construct, use,
because they ‘conjectured that the underlying success
construct that researchers have been trying to tap is
Usefulness, not Use’ (p. 93). Seddon and Kiew’s concept
of usefulness is equivalent to the idea of perceived
usefulness in TAM by Davis (1989). They argued that,
for voluntary systems, use is an appropriate measure;
however, if system use is mandatory, usefulness is a better
measure of IS success than use. DeLone & McLean (2003)
responded that, even in mandatory systems, there can
still be considerable variability of use and therefore the
variable use deserves to be retained.
Researchers have also suggested that service quality be
added to the D&M model. An instrument from the
marketing literature, SERVQUAL, has become salient
within the IS success literature within the past decade.
SERVQUAL measures the service quality of IT departments,
as opposed to individual IT applications, by measuring and
comparing user expectations and their perceptions of the
IT department. Pitt et al. (1995) evaluated the instrument
from an IS perspective and suggested that the construct of
service quality be added to the D&M model. Some
researchers have resisted this change (Seddon, 1997),
while others have endorsed it (Jiang et al., 2002). DeLone
& McLean (2003), after reviewing and evaluating this
debate, decided to add service quality in their updated IS
success model stating that ‘the changes in the role of IS
over the last decade argue for a separate variable – the
‘‘service quality’’ dimension’ (p. 18).
Another well-known proposed modification to the
D&M model is the changes offered by Seddon (1997).
He argued that the D&M model in its original form was
confusing, partly because both process and variance
models were combined within the same framework.
While he claimed that this was a shortcoming of the
model, DeLone & McLean (2003) responded that they
believed that this was one of its strengths, with the
insights provided, respectively, by process and variance
models being richer than either is alone. Seddon
furthered suggested that the concept of use is highly
ambiguous and suggested that further clarification was
needed to this construct. He derived three different
potential meanings for the use construct, as well as
parsing out the process and variances portions of the
model. The D&M model of IS success was intended to be
‘both complete and parsimonious’; however, the changes
introduced by Seddon complicates the model, thereby
reducing its impact.
In addition to the modifications proposed by Seddon,
there have been other calls to revise or extend the model.
Some researchers have modified it to evaluate success of
specific applications such as knowledge management
(e.g., Jennex & Olfman, 2002; Kulkarni et al., 2006; Wu &
Wang, 2006) and e-commerce (e.g., Molla & Licker, 2001;
DeLone & McLean, 2004; Zhu & Kraemer, 2005). Other
researchers have made more general recommendations
concerning the model (e.g., Ballantine et al., 1996).
Recognizing these proposed modifications to their
model, D&M, in a follow-up work, reviewed empirical
studies that had been performed during the years since
1992 and revised the original model accordingly (DeLone
& McLean, 2002, 2003). The updated model is shown in
Figure 2.
This updated IS success model accepted the Pitt et al.
(1995) recommendation to include service quality as a
construct. Another update to the model addressed the
criticism that an information system can affect levels
other than individual and organizational levels. Because
IS success affects workgroups, industries, and even
societies (Myers et al., 1997; Seddon et al., 1999), D&M
replaced the variables, individual impact and organizational
impact, with net benefits, thereby accounting for
benefits at multiple levels of analysis. This revision
allowed the model to be applied to whatever level of
analysis the researcher considers most relevant.
A final enhancement made to the updated D&M model
was a further clarification of the use construct. The
authors explained the construct as follows: ‘Use must
precede ‘‘user satisfaction’’ in a process sense, but positive
experience with ‘‘use’’ will lead to greater ‘‘user satisfaction’’
in a causal sense’ (DeLone & McLean, 2003). They
went on to state that increased user satisfaction will lead
to a higher intention to use, which will subsequently
affect use.
The D&M model has also been found to be a useful
framework for organizing IS success measurements. The
model has been widely used by IS researchers for understanding
and measuring the dimensions of IS success.
Furthermore, each of the variables describing success of
an information system was consistent with one or more
of the six major success dimensions of the updated
model. The dimensions of success include:
 System quality – the desirable characteristics of an
information system. For example: ease of use, system
System
Quality
Information
Quality
Service
Quality
Use
User
Satisfaction
Net Benefits
Intention
to Use
Figure 2 Updated DeLone and McLean IS success model
(2003).
Measuring information systems success Stacie Petter et al238
European Journal of Information Systems
flexibility, system reliability, and ease of learning, as
well as system features of intuitiveness, sophistication,
flexibility, and response times.
 Information quality – the desirable characteristics of the
system outputs; that is, management reports and Web
pages. For example: relevance, understandability, accuracy,
conciseness, completeness, understandability,
currency, timeliness, and usability.
 Service quality – the quality of the support that system
users receive from the IS department and IT support
personnel. For example: responsiveness, accuracy,
reliability, technical competence, and empathy of the
personnel staff. SERVQUAL, adapted from the field of
marketing, is a popular instrument for measuring IS
service quality (Pitt et al., 1995).
 System use – the degree and manner in which staff and
customers utilize the capabilities of an information
system. For example: amount of use, frequency of use,
nature of use, appropriateness of use, extent of use, and
purpose of use.
 User satisfaction – users’ level of satisfaction with
reports, Web sites, and support services. For example,
the most widely used multi-attribute instrument for
measuring user information satisfaction can be found
in Ives et al. (1983).
 Net benefits – the extent to which IS are contributing to
the success of individuals, groups, organizations,
industries, and nations. For example: improved decision-making,
improved productivity, increased sales,
cost reductions, improved profits, market efficiency,
consumer welfare, creation of jobs, and economic
development. Brynjolfsson et al. (2002) have used
production economics to measure the positive impact
of IT investments on firm-level productivity.
The practical application of the D&M model is
naturally dependent on the organizational context. The
researcher wanting to apply the D&M model must have
an understanding of the information system and organization
under study. This will determine the types of
measures used for each success dimension. The selection
of success dimensions and specific metrics depend on the
nature and purpose of the system(s) being evaluated. For
example, an e-commerce application would have some
similar success measures and some different success
measures compared to an enterprise system application.
Both systems would measure information accuracy, while
only the e-commerce system would measure personalization
of information. An information system that is
managed by a vendor will measure the service quality
of the vendor, rather than of the IS department. Seddon
et al. (1999) developed a context matrix that is a valuable
reference for the selection of success measures based on
stakeholders and level of analysis (individual application
or IS function). Ideally, the D&M model is applicable in a
variety of contexts; however, the limits of the model are
not well-known or understood. This research examines
one of the potential boundary conditions for the model
and identifies areas that warrant additional attention.
The current understanding of IS success
There have been a number of studies that have attempted
to further the understanding of the D&M model by
attempting to validate some, or all, of the entire model in
a single study. Seddon & Kiew (1996) examined the
relationships among four of the constructs and found
good support. Rai et al. (2002) compared the original
D&M model (1992) to the respecified model created by
Seddon (1997) and found that the D&M model stood up
reasonably well to the validation attempt and outperformed
the Seddon model. Sedera et al. (2004) also
recently tested several success models, including the
D&M and Seddon models, against empirical data and
determined that the DeLone & McLean Model provided
the best fit for measuring enterprise systems success.
McGill et al. (2003) examined the full model, but found
four paths in the original IS success model insignificant
(system quality-use, information quality-use, intended
use-individual impact, and individual impact-organizational
impact).
Some researchers (e.g., Au et al., 2002; DeLone &
McLean, 2003; Grover et al., 2003) have conducted
literature reviews to examine if the results of empirical
studies support the relationships posited by the original
success model. These literature reviews reveal that some
relationships within the model have received consistent
support (i.e., significant results across all studies) while
others have received only mixed support (i.e., some
studies find significant results while others are nonsignificant).
Other researchers have performed metaanalyses
to examine one or more relationships in the
D&M model (e.g., Mahmood et al., 2001; Bokhari, 2005;
Sabherwal et al., 2006). The most comprehensive metaanalysis
examining the D&M model was performed by
Sabherwal et al. (2006). Sabherwal et al.’s work has been
instrumental in synthesizing the quantitative research
related to IS success and has validated a substantial
portion of the D&M model.
This study extends Sabherwal et al.’s work in several
ways. First, by performing a qualitative literature review,
studies that use qualitative methods or do not report
enough information to be included in a meta-analysis
can be included in this analysis of the literature. Second,
by not aggregating each study into a single numerical
value, it is possible to examine issues associated with
measuring the various constructs within the D&M model
and examine the lack of consistency among studies in
terms of context and constructs. Third, this work
examines if the level of analysis under study (i.e., the
individual or the organization) is a boundary condition
for the D&M model.
Methodology
One of the most well-established methods to integrate
research findings and assess the cumulative knowledge
Measuring information systems success Stacie Petter et al 239
European Journal of Information Systems
within a domain is a qualitative literature review (Oliver,
1987). This method allows a researcher to analyze and
evaluate both quantitative and qualitative literature
within a domain to draw conclusions about the state of
the field. As with any research technique, there are
limitations. The primary limitation with this approach is
that when conflicting findings arise, it becomes difficult
to determine the reason for the conflicting results. Some
also perceive that because the literature review is
qualitative, it is subjective in nature and provides little
‘hard evidence’ to support a finding.
To counter these shortcomings, the research technique
of meta-analysis has become quite popular in the social
sciences and now in IS. Meta-analysis is an interesting
and useful technique to synthesize the literature using
quantitative data reported across research studies. The
result of a meta-analysis is an ‘effect size’ statistic that
states the magnitude of the relationship and whether or
not the relationship between variables is statistically
significant (Oliver, 1987; Hwang et al., 2000). This
approach too has its limitations. A key limitation is the
need to exclude studies that use qualitative techniques to
examine success or studies that fail to report the
information required for the statistical calculations for
the meta-analysis. While the meta-analysis produces a
quantified result regarding the relationship between two
variables, the need to exclude some studies may not
present a complete picture of the literature. Furthermore,
a meta-analysis does not examine the direction of
causality, because the effect size is an adjusted correlation
between two variables.
There have been meta-analyses examining one or more
of the elements of IS success (Hwang et al., 2000;
Mahmood et al., 2001; Bokhari, 2005; Sabherwal et al.,
2006); therefore, this paper seeks to obtain a different,
qualitative view of the literature to answer a different set
of research questions. While a meta-analysis is aimed at
answering the question: ‘Is there a correlation between
two variables?’, a qualitative literature review is better
equipped to explain how the relationships have been
studied in the literature, if there appears to be support for
a causal relationship between two variables, and examines
if there are any potential boundary conditions for
the model.
Scope of the literature search
To find research that has been published on IS success,
full-text searches in numerous online databases (EBSCO
Host, ABI Inform, and Web of Knowledge) were performed
using multiple keywords, such as ‘IS success,’ ‘IS
effectiveness,’ ‘DeLone and McLean,’ etc. Print issues
of well-known IS journals unavailable electronically
were also examined to ensure that applicable studies
were included.
As a means to ensure that the bibliography of relevant
studies was complete, the list of studies was triangulated
with the reference lists of several papers and Web sites
that examined the history of IS success, such as the
updated DeLone & McLean paper (2003) and an AISWorld
Web site devoted to IS effectiveness (Grover et al.,
2003). A total of 180 empirical and conceptual papers
were identified in this wide-ranging search for IS success
research (see Appendix A for a list of studies examined).
These papers were published in the time period between
1992, the year the D&M success model was first
published, and 2007. From this collection of papers, only
papers reporting empirical results (both quantitative and
qualitative research) of interrelationships among the
D&M success dimensions are included in this paper; this
yielded a total of 90 papers.
To perform a literature review, it is necessary to
examine as much related literature as possible on the
topic; however, to prevent from being overwhelmed, the
focus of this research is on utilitarian IS that can be used
by organizations or individuals to improve performance.
Also, given that many other reference disciplines also
study IS (e.g., marketing, psychology, management, etc.),
the primary searches for literature focused on journals
within the IS discipline. However, we did not restrict the
literature review to a specific type of information system
or a specific use context (i.e., individual vs organizational
or voluntary vs mandatory).
Organizing the literature review
Because of the popularity of the D&M model in the
academic literature, it seemed appropriate to organize the
studies of IS success that were found using its taxonomy.
The findings from empirical studies on IS success are
organized by success constructs or dimensions. Subsections
are grouped based on the expected causal relationships
between paired success constructs. Organizing the
literature in this manner helps to examine whether there
is support for each of the proposed relationships within
the D&M model (Figure 2). Table 1 lists each of the 15
causal relationships evaluated in this manuscript.
The subsections are further organized by the unit of
analysis, whether individual or organizational stakeholders
are the focus of the research. The updated D&M
model suggests that IS success can be examined at
different levels (DeLone & McLean, 2002, 2003); therefore,
this literature review investigated if there are
differences in the strengths of the relationships based
on whether the studies focused on an individual or
organizational level when measuring and evaluating the
various success constructs and relationships.
Each of the constructs of the D&M model has multiple
operationalizations; and the support, or lack of support,
for relationships between constructs may be due to the
manner in which the constructs were measured. Therefore,
this review also discusses the specific success
measures that were used in the selected studies.
Literature review of IS success studies
Measuring the six constructs of IS success
There are many approaches to measuring success
of IS. Some researchers have developed approaches to
Measuring information systems success Stacie Petter et al240
European Journal of Information Systems
measuring success in specific industries by incorporating
the various dimensions of the D&M model (Weill &
Vitale, 1999; Skok et al., 2001). However, there are many
scales that have been used to measure the dimensions of
the IS success model individually, with some being more
thorough than others. This section identifies some of the
different operationalizations of each construct.
Measuring system quality Perceived ease of use is the
most common measure of system quality because of the
large amount of research relating to the TAM (Davis,
1989).
However, perceived ease of use does not capture the
system quality construct as a whole. Rivard et al. (1997)
developed and tested an instrument that consists of 40
items that measure eight system quality factors: namely,
reliability, portability, user friendliness, understandability,
effectiveness, maintainability, economy, and verifiability.
Others have created their own indexes of system
quality using the dimensions identified by D&M in their
original model (Coombs et al., 2001) or via their own
review of the system quality literature (Gable et al., 2003).
Measuring information quality Information quality is
often a key dimension of end-user satisfaction instruments
(Ives et al., 1983; Baroudi & Orlikowski, 1988; Doll
et al., 1994). As a result, information quality is often not
distinguished as a unique construct but is measured as a
component of user satisfaction. Therefore, measures of
this dimension are problematic for IS success studies.
Fraser & Salter (1995) developed a generic scale of
information quality, and others have developed their
own scales using the literature that is relevant to the type
of information system under study (Coombs et al., 2001;
Wixom & Watson, 2001; Gable et al., 2003).
Measuring service quality As discussed earlier, there is a
debate on the validity of SERVQUAL as a service quality
measure (Pitt et al., 1995; Kettinger & Lee, 1997; Van
Dyke et al., 1997). While SERVQUAL is the most
frequently used measure for service quality in IS, it has
received some criticism. However, using confirmatory
factor analysis, Jiang et al. (2002) found that SERVQUAL
is indeed a satisfactory instrument for measuring IS
service quality.
Other measures of service quality have included the
skill, experience, and capabilities of the support staff
(Yoon & Guimaraes, 1995). With the growing popularity
of outsourcing for systems development and support,
service quality often involves an external provider. The
responsiveness of the vendor affects the perception of
how ‘cooperative’ that vendor will be (Gefen, 2000).
Measuring use Empirical studies have adopted multiple
measures of IS use, including intention to use, frequency
of use, self-reported use, and actual use. These different
measures could potentially lead to mixed results between
use and other constructs in the D&M model. For
example, research has found a significant difference
between self-reported use and actual use (Collopy, 1996;
Payton & Brennan, 1999). Typically, heavy users tend to
underestimate use, while light users tended to overestimate
use. This suggests that self-reported usage may
be a poor surrogate for actual use of a system. Yet,
Venkatesh et al. (2003), for example, found a significant
relationship between intention to use and actual usage.
In addition, frequency of use may not be the best way
to measure IS use. Doll & Torkzadeh (1998) suggest that
more use is not always better and they developed an
instrument to measure use based on the effects of use,
rather than by frequency or duration. Burton-Jones &
Straub (2006) have reconceptualized the systems usage
construct by incorporating the structure and function of
systems use.
Others have suggested the need to examine use from a
multilevel perspective across the individual and organizational
levels to enable a better understanding of this
construct (Burton-Jones & Gallivan, 2007).
Measuring user satisfaction The most widely used user
satisfaction instruments are the Doll et al. (1994) EndUser
Computing Support (EUCS) instrument and the Ives
et al. (1983) User Information Satisfaction (UIS) instrument.
In a comparison between Doll and Torkzadeh’s
EUCS and Ives’ et al. UIS, Seddon & Yip (1992) found the
EUCS instrument outperformed the UIS instrument in
the context of accounting IS. However, both the EUCS
Table 1 Proposed success relationships posited in D&M
model (2003)
System quality - System usea
System quality - User satisfaction
System quality - Net benefits
Information quality - System use
Information quality - User satisfaction
Information quality - Net benefits
Service quality - System use
Service quality - User satisfaction
Service quality - Net benefits
System use - User satisfaction
System use - Net benefits
User satisfaction - System use
User satisfaction - Net benefits
Net benefits - System use
Net benefits - User satisfaction
a
We chose to consider both intention to use and other measures of
system use as the same construct for this literature review. Although
D&M did distinguish between intention to use and system use in their
updated model, intention to use is generally an individual level construct.
This is not a concept that is consistent with studies employing an
organizational unit of analysis. Furthermore, by parsing the use
constructs into two separate subconstructs (i.e., intention to use and
use), it makes an already complex paper (with 15 pairwise relationships)
even more complex (adding at least six pairwise relationships to the
analysis). In the discussion of the results of the literature review, we do
identify those studies that measure intention to use as opposed to other
measures of system use.
Measuring information systems success Stacie Petter et al 241
European Journal of Information Systems
and UIS instruments contain items related to system
quality, information quality, and service quality, rather
than only measuring overall user satisfaction with the
system. Because of this, some researchers have chosen to
parse out the various quality dimensions from these
instruments and either use a single item to measure
overall satisfaction with an information system (Rai et al.,
2002) or use a semantic differential scale (Seddon & Yip,
1992). Others have used scales for attitude that are
compatible with the concept of user satisfaction (Coombs
et al., 2001).
Measuring net benefits There are an abundance of
methods to measure net benefits at both the individual
and organizational level of analysis. Perceived usefulness
or job impact is the most common measure at the
individual level. Yet, there have been occasional problems
with the perceived usefulness items (e.g., Adams
et al., 1992). Segars & Grover (1993) analyzed the data
from the Adams et al. study using confirmatory factor
analysis and eliminated an item ‘works more quickly’ in
the usefulness construct. In addition, the authors found
that ‘job performance’ and ‘effectiveness’ did not fit well
with perceived usefulness. The authors used these two
items to measure a separate construct called effectiveness.
This three-factor construct, perceived ease of use, perceived
usefulness, and effectiveness, resulted in a relatively
strong fit, as opposed to the poor fit obtained with
the original TAM model. Torkzadeh & Doll (1999) have
created an instrument to measure different aspects of
impact – task productivity, task innovation, customer
satisfaction, and management control – to augment their
EUCS instrument.
At the organizational level, a variety of measures are
employed; but profitability measurements seem to be
preferred. The multiple measures for net benefits at each
level of analysis make it more difficult to interpret the
relationship among some of the success constructs and
net benefits. In some studies, the lack of significant
findings may be an artifact of measurement, the type of
system studied, or some other factor. A key point in terms
of measuring organizational benefits, however, is that
researchers must ensure that the person evaluating
organizational benefits is in a position to answer the
questions. Asking users of a system to assess the improved
profitability due to the system may not be the best
approach. Asking senior managers or referring to objective
data from annual reports may be more appropriate
when trying to measure organizational benefits.
A comprehensive IS success measurement
instrument
Sedera et al. (2004) have developed and validated a multidimensional
IS success instrument for enterprise systems.
This success instrument has been applied and tested in
three separate studies. It consists of four dimensions –
system quality, information quality, individual impact,
and organizational impact – and 27 item measures: nine
measures of system quality, six measures of information
quality, four measures of individual impact, and eight
measures of organizational impact (see Table 2).
What makes this particular instrument to measure IS
success unique is that this instrument captures the
multidimensional and complex nature of IS success by
measuring four key success dimensions and by using at
least four measures for each dimension. The instrument
has strong construct validity in that it captures multiple
aspects of each variable, which is a dramatic change from
much of the measurement of IS success constructs that
focus on only one aspect of the construct. Another
strength of this model is that the instrument was
rigorously tested within the context of enterprise systems
to ensure its validity.
An interesting finding from this research by Sedera
et al. is that user satisfaction was eliminated from their
success measurement model because it added little
explanatory power after the primary four constructs.
Use was also eliminated because the system under study
was mandatory causing little measurable variation in use.
It is encouraging to see research conducted to create a
strong, multidimensional instrument to measure IS
success, which overcomes a major shortcoming in
previous IS empirical work; namely, inadequate measurement
of the dependent variable, IS success. It would be
interesting to see if this instrument is relevant to other
types of IS beyond enterprise systems. Further research
Table 2 Validated measures for IS success (Sedera et al., 2004) (used by permission)
System quality Information quality Individual impact Organizational impact
Ease of use Availability Learning Organizational costs
Ease of learning Usability Awareness/recall Staff requirements
User requirements Understandability Decision effectiveness Cost reduction
System features Relevance Individual productivity Overall productivity
System accuracy Format Improved outcomes/outputs
Flexibility Conciseness Increased capacity
Sophistication e-Government
Integration Business process change
Customization
Measuring information systems success Stacie Petter et al242
European Journal of Information Systems
examining to see if use and user satisfaction provide
additional explanatory value in different settings, particularly
for voluntary systems, would also provide more
insight into the measurement of IS success.
Fifteen pairwise comparisons of the IS success
constructs
The 15 pairs of relationships shown in Table 1 are
discussed in the following subsections. The studies
included in this literature review are related to a variety
of industries and variety of IS. The only requirements for
inclusion in this review of the literature is that the
manuscript must (a) report an empirical result (i.e., either
quantitative or qualitative) and (b) examine a relationship,
broadly defined, within the D&M model. There
were no restrictions based on industry, type of information
system, mandatory or voluntary systems nature of
the system, to better ascertain limitations and boundary
conditions for the D&M model.
System quality-use There is mixed support for this
relationship at the individual level of analysis within the
literature. Many studies measure system quality as
perceived ease of use and find positive relationships with
various operationalizations of use in a variety of systems
at the individual level of analysis. Perceived ease of use is
related to system dependence (Rai et al., 2002; Kositanurit
et al., 2006), behavioral intentions to use the system
(Venkatesh & Davis, 2000; Venkatesh & Morris, 2000;
Hong et al., 2001/2002), extent of use (Hsieh & Wang,
2007) and self-reported use (Adams et al., 1992). Yet other
research has found that perceived ease of use is only
weakly related to actual use (Straub et al., 1995) and is not
significantly related to intention to use (Subramanian,
1994; Agarwal & Prasad, 1997; Lucas & Spitler, 1999;
McGill et al., 2003; Klein, 2007), self-reported use (Straub
et al., 1995; Gefen & Keil, 1998; Lucas & Spitler, 1999),
and system dependence (Goodhue & Thompson, 1995).
One study even found, for complex applications such as
Lotus 1–2–3, perceived ease of use was negatively related
to system use (Adams et al., 1992), suggesting that both
system quality and use are complex constructs.
Other research on system quality, using measures
besides perceived ease of use, also obtained mixed results.
For example, Iivari (2005) found a positive relationship
between system quality and use. Goodhue & Thompson
(1995) found a significant relationship between reliability
and system dependence. Another study identified a
significant relationship between perceived ease of use
and system usage as measured by the number of different
applications used, number of computer-supported business
tasks, duration, and frequency of use at the
organizational level (Igbaria et al., 1997). Suh et al.
(1994) reported a significant correlation between performance
of an information system (perceived ease of use,
accuracy, etc.) and frequency of use and system dependence.
In a study to determine which characteristics of an
information system affect intention to use and actual use,
Agarwal & Prasad (1997) found mixed results when
examining different aspects of system quality, such as
relative advantage and compatibility. Venkatesh et al.
(2003) found a significant relationship between effort
expectancy and intentions to use the system in both
voluntary and mandatory settings when measured one
month after implementation of a new information
system. However, this relationship became non-significant
when measured three and six months after the
implementation of the system. A case study reporting
both qualitative (Markus & Keil, 1994) and quantitative
(Gefen & Keil, 1998) results found, however, that
perceived system quality did not guarantee the usage of
the system. Kositanurit et al. (2006) determined that
reliability of an ERP system does not have an effect on
utilization of the system by individual users.
Examining the relationship between system quality
and use at the organizational level found mixed support
for this relationship as well. Caldeira & Ward (2002), in
their study of small- and medium-sized Portuguese
manufacturing enterprises (SMEs), identified the quality
of available software in the market as a factor related to IS
adoption and success. One study found that the
perceived ease of use of a manufacturing resource
planning system did not significantly affect self-reported
use (Gefen, 2000). Another study, examining factors
related to expert system longevity and use in an
organization, noted that technical reasons, such as
system quality, were not the main consideration for use
or discontinuance of use, which offers further support
that system quality may not be a good predictor of use
(Gill, 1995). However, a study of the turnaround of the
notorious London Ambulance Service Dispatch System
failure found that improved system quality was positively
related to subsequent system use (Fitzgerald & Russo,
2005). A study looking at IS at a single site using
responses from multiple users found that the technical
quality of the system was negatively related to use (Weill &
Vitale, 1999). Weill and Vitale assumed that this counterintuitive
result was probably due to the fact that systems
that are heavily used are often fixed quickly, without
adequate testing and integration with the current system.
This may affect the perception of the technical quality of
a system. Premkumar et al. (1994) did not find that the
complexity of a system affects the initial use and
adoption of an EDI system; however, the technical
compatibility of the system with existing hardware and
software did affect initial use and adoption of an EDI
system.
System quality-user satisfaction At the individual unit
of analysis, there is strong support for the relationship
between system quality and user satisfaction (Iivari,
2005). Several types of IS have been examined, and the
type of information system affects how some researchers
measure system quality. For example, the functionality of
a management support information system, which is one
measure of system quality, has been found to be
Measuring information systems success Stacie Petter et al 243
European Journal of Information Systems
significantly related to user satisfaction (Gelderman,
2002). For knowledge management systems, system
quality was also found to be strongly related to user
satisfaction (Kulkarni et al., 2006; Wu & Wang, 2006;
Halawi et al., 2007). For Web sites, system quality,
measured as reliability and download time, is significantly
related to user satisfaction in two different studies
(Kim et al., 2002; Palmer, 2002). Perceived ease of use also
has a significant relationship to user satisfaction (Devaraj
et al., 2002; Hsieh & Wang, 2007). Researchers have also
examined more general IS and found a strong relationship
between system quality and user satisfaction using a
variety of measures and IS (Seddon & Yip, 1992; Yoon
et al., 1995; Guimaraes et al., 1996; Seddon & Kiew, 1996;
Bharati, 2002; Rai et al., 2002; McGill et al., 2003;
Almutairi & Subramanian, 2005; McGill & Klobas, 2005;
Wixom & Todd, 2005). A case study found necessary, but
not sufficient, relationships between system quality and
user satisfaction and ease of use and user satisfaction
(Lexlercq, 2007).
At the organizational level, few studies have examined
the relationship between system quality and user satisfaction.
Therefore, it is difficult to draw any conclusions on
this relationship at this particular level of analysis. One
study found that the functionality of executive IS is
significantly related to user satisfaction (Benard & Satir,
1993). In two longitudinal case studies, Scheepers et al.
(2006) identified a relationship between ease of use of a
mobile computing information system and user satisfaction.
Premkumar et al. (1994) found no relationship
between the complexity of a system and user satisfaction.
System quality-net benefits The relationship between
system quality and net benefits has moderate support
within the literature. In general, there is a positive impact
on individual performance, although the relationship
between perceived ease of use as a measure of system
quality and perceived usefulness has seen mixed results.
Some studies have a found a significant relationship
(Adams et al., 1992; Gefen & Keil, 1998; Agarwal &
Prasad, 1999; Lucas & Spitler, 1999; Venkatesh & Davis,
2000; Venkatesh & Morris, 2000; Hong et al., 2001/2002;
Devaraj et al., 2002; Yang & Yoo, 2004; Wixom & Todd,
2005; Hsieh & Wang, 2007), while others have found no
significant association (Subramanian, 1994; Chau & Hu,
2002; Kulkarni et al., 2006; Wu & Wang, 2006). Seddon &
Kiew (1996) and Shih (2004) found that system quality is
significantly related to perceived usefulness. Systems
reliability and perceived ease of use had no impact on
productivity and effectiveness (Goodhue & Thompson,
1995). McGill & Klobas (2005) found no relationship
between system quality and individual impact as measured
by decision-making quality and productivity.
Kositanurit et al. (2007) identified a significant relationship
between perceived ease of use and performance, but
no relationship between reliability and performance for
individual users of ERP systems. Bharati & Chaudhury
(2006) found a significant relationship between system
quality, measured by reliability, flexibility, ease of use,
and convenience of access, to decision-making satisfaction
in an e-commerce environment.
At the organizational level, there exists strong support
for the relationship of system quality to net benefits. The
quality of an EDI system was found to be related to
organizational efficiency, sales, and organizational image
(Farhoomand & Drury, 1996). System quality of a data
warehouse was associated with decreased time and effort
for decision making (Wixom & Watson, 2001). Gefen
(2000) also found that perceived ease of use and
perceived correctness of software were related to perceived
usefulness. The technical performance of an
information system was found to indirectly affect the
perceived value of the system, mediated by use and user
satisfaction (Weill & Vitale, 1999). Another study compared
system quality and impact of system use at
operational, tactical, and strategic levels (Bradley et al.,
2006). The relationship between system quality and
impact of use at these various levels was significant.
However, when these results were analyzed more closely,
it was found that this relationship was not significant at
all for formal firms, and only significant at operational
levels within entrepreneurial firms.
Information quality-use Few studies have examined the
relationship between information quality and use at both
the individual and organizational levels. One reason for
this is that information quality tends to be measured as a
component of user satisfaction measures, rather than
being evaluated as a separate construct. Most of the
studies that have examined the relationship between
information quality and use focused on IS success models
as a whole. Rai et al. (2002) found that information
quality is significantly related to use, when use is
measured by system dependence. A study of knowledgemanagement
systems found that information (or knowledge)
quality was significantly related to intention to use
(Halawi et al., 2007). Yet, two studies found that
information quality is not significantly related to intention
to use (McGill et al., 2003; Iivari, 2005). In their
study of task–technology fit, Goodhue & Thompson
(1995) found that the quality, locatability, authorization,
and timeliness of information were not significantly
related to utilization, as measured by system dependence,
yet compatibility of information was related to system
dependence.
At the organizational level, Fitzgerald & Russo (2005),
in their study of the London Ambulance Dispatch
System, found a positive relationship between information
quality and system use.
Information quality-user satisfaction The relationship
between information quality and user satisfaction is
strongly supported in the literature (Iivari, 2005; Wu &
Wang, 2006). Studies have found a consistent relationship
between information quality and user satisfaction at
the individual unit of analysis (Seddon & Yip, 1992;
Measuring information systems success Stacie Petter et al244
European Journal of Information Systems
Seddon & Kiew, 1996; Bharati, 2002; Rai et al., 2002;
McGill et al., 2003; Almutairi & Subramanian, 2005;
Wixom & Todd, 2005; Kulkarni et al., 2006; Chiu et al.,
2007; Halawi et al., 2007). Studies specifically examining
the information quality aspects of Web sites, such as
content and layout, have found significant relationships
between these constructs and user satisfaction (Kim et al.,
2002; Palmer, 2002). Marble (2003), however, did not
find a significant relationship between measures of
information quality and user satisfaction of two organizational
IS examined in his study.
At the organizational level of analysis, support also
exists for the effect of information quality on user
satisfaction, but there are not enough studies examining
this relationship to reach a strong conclusion. In a
qualitative study on system success, data quality and
user satisfaction, measured by user attitudes, were found
to be directly related to one another (Coombs et al.,
2001). Another qualitative case study identified multiple
comments from respondents suggesting an association
between information quality (i.e., content, accuracy,
timeliness, and format) and user satisfaction (Scheepers
et al., 2006). A quantitative study also found a significant
link between information quality and managerial satisfaction
of hardware, software, and support of an
information system (Teo & Wong, 1998).
Information quality-net benefits There is moderate
support for the positive impact of information quality
on individual performance. Gatian (1994) found that
information quality was related to decision-making
efficiency. Information quality has also been found to
be associated with quality of work and time savings
(D’Ambra & Rice, 2001; Shih, 2004) and decision-making
satisfaction (Bharati & Chaudhury, 2006). Perceived
information quality was also significantly related to
perceived usefulness (i.e., a net benefit) (Kraemer et al.,
1993; Seddon & Kiew, 1996; Rai et al., 2002; Shih, 2004;
Wu & Wang, 2006). Kositanurit et al. (2006) discovered a
significant relationship between information quality and
performance among users of ERP systems. However, in
the context of a knowledge management system, perceived
content quality was not directly related to
perceived usefulness (Kulkarni et al., 2006). A study of
digital libraries found that the relevance of the information
retrieved had a significant effect on perceived
usefulness, yet the clarity of the terminology used and
screen design of the content presented had no relationship
with perceived usefulness (Hong et al., 2001/2002).
The relationship between information quality and
benefits at the organizational level has shown mixed
results, depending on how net benefits are measured. Yet
again, more research is needed to reach a conclusion in
terms of this relationship. Information quality was found
to be significantly related to better perceptions of the
work environment (i.e., morale, job content, interesting
work) (Teo & Wong, 1998) and to organizational
efficiency, sales, and organizational image (Farhoomand
& Drury, 1996). Data quality was directly related to
perceived decrease in time and effort for decision making
in Wixom & Watson’s (2001) study. On the other hand,
information quality was not found to be significantly
related to organizational impact as measured by productivity,
competitiveness, and management improvement
(Teo & Wong, 1998). Bradley et al. (2006) also studied
information quality and the impact of system use in
formal and entrepreneurial firms and found largely nonsignificant
results.
Service quality-use There is little literature that examines
the relationship between service quality and use at
the individual or organizational level. One study examining
this relationship examined accounting IS in Korean
firms (Choe, 1996). In this study, the number of years of
experience of the IS support personnel was weakly related
(Po0.1) to frequency and willingness of use. When
analyzed further using Nolan’s (1973) Stage Model to
measure the maturity of an information system, years of
experience of the IS support personnel was significantly
correlated with use; however, in later stages of maturity,
IS personnel experience was found to be negatively
correlated (although not significantly) with usage. This
same study examined the role of user training and
education and use and found a non-significant relationship
between frequency and willingness of use. Again,
analyzing the data further using Nolan’s Stage Model to
determine the maturity of the system, user training and
education were significantly related to use in the earlier
stages of the information system, but not in the later
stages. In another study, documentation of a system was
not a predictor of utilization in a survey of ERP users
(Kositanurit et al., 2006). A study of knowledge-management
systems found that service quality did not predict
intention to use (Halawi et al., 2007).
At the organizational level, in the study of the London
Ambulance System, the effective role of the technical
staff (i.e., service quality) was positively related to the
eventual use of the system (Fitzgerald & Russo, 2005). In a
study of expert systems, the retention of service staff (and
the related funding) to maintain an expert system was a
major factor in determining the longevity of the system.
Caldeira & Ward (2002), in their study of Portuguese
SMEs, found that competency of the support staff, vendor
support, and availability of training affected use and
adoption of IS.
Service quality-user satisfaction Several studies have
examined the relationship between service quality and
user satisfaction; however, the findings of these studies
suggest mixed support for this relationship. Researchers
have measured service quality using multiple methods,
which may account for the inconsistent findings. Some
researchers have looked at service quality by examining
the characteristics of the support personnel; however,
examining the relationship between personnel characteristics
and user satisfaction has produced mixed results.
Measuring information systems success Stacie Petter et al 245
European Journal of Information Systems
Choe (1996) found that IS personnel experience does not
significantly affect user satisfaction of accounting IS in
Korean firms. Additional analysis of these data, however,
noted that if the system was recently implemented, the
experience of the IS support personnel was slightly
correlated with user satisfaction; yet during later stages
of the information system, there was a non-significant
relationship between years of experience and user
satisfaction of the support team. Another study found
that the technical performance of the developers (based
on their responsiveness to problems) was positively
related to user satisfaction (Leonard-Barton & Sinha,
1993). Yoon et al. (1995) had a similar result in that
developer skill had a significant effect on user satisfaction
of expert systems. A case study performed by Leclercq
(2007) found that the relationship between the IS
function and the users as well as the quality of support
and services provided by the IS function had an impact
on user satisfaction. The mutual understanding between
the IS group and the users during the implementation of
a project did not have significant impact on satisfaction
of the resulting system (Marble, 2003). Chiu et al. (2007)
examined the role of support on user satisfaction in an elearning
environment and found a non-significant
relationship. Choe (1996) also examined the role of
training and education on user satisfaction of an
information system and found no significant relationship
at any stage of IS implementation.
Examining service quality more broadly, rather than
just in terms of personnel and training, there is still
mixed support for its effect on user satisfaction. Using the
SERVQUAL instrument, which examines the expectations
and perceptions that users have on service quality,
Kettinger & Lee (1994) found that service quality is
positively and significantly related to user satisfaction of
information services in a survey of undergraduate
students rating the university’s computing services
department. Another study of university support services
found a relationship between service quality and user
satisfaction and identified software upgrades, staff response
time, and documentation of training materials as
the service quality factors having the most influence on
user satisfaction (Shaw et al., 2002). Although the two
studies in university settings found support for this
relationship, a study of 31 government organizations
examining internal computing support and user satisfaction
did not find a significant relationship (Aladwani,
2002). A study of Web sites found that responsiveness of
the site in terms of feedback, assistance, and frequently
asked questions was not related to user satisfaction of the
Web site (Palmer, 2002). In another Web setting using the
SERVQUAL measure, the empathy and assurance aspects
of service quality were related to user satisfaction, but not
to reliability or responsibility (Devaraj et al., 2002).
Halawi et al. (2007) found a significant relationship
between service quality, measured using SERVQUAL,
and user satisfaction in a knowledge-management context.
These findings suggest the sensitivity of the
construct of service quality to the manner in which it is
measured.
At the organizational level, more research is clearly
needed. A qualitative study of system success found that
higher quality training and friendly IS support staff led to
more positive attitudes about the system (Coombs et al.,
2001). Other researchers have examined service quality in
terms of vendor support. The role of outside support on
user satisfaction has also yielded mixed results. One study
found that higher levels of consultant effectiveness and
higher levels of vendor support created higher levels of
user satisfaction (Thong et al., 1996). In a study attempting
to determine the outside support method that yielded
higher levels of satisfaction, Thong et al. (1994) found
that organizations working solely with a vendor when
implementing an IS were more satisfied than organizations
using both a vendor and a consultant. Another
study examining the use of consultants for selecting and
implementing an executive information system found a
negative (although not significant) relationship with user
satisfaction; the authors suggested that this counterintuitive
finding was due to higher expectations that
arose when using consultants (Benard & Satir, 1993).
Service quality-net benefits The relationship between
service quality and net benefits has moderate support in
the individual context. Igbaria et al. (1997) found that
external computing support was related to perceived
system usefulness, but that internal computing support
was not related to perceived usefulness. Perceived developer
responsiveness (Gefen & Keil, 1998) and user
training provided by the internal computing department
(Igbaria et al., 1997; Agarwal & Prasad, 1999) have been
found to be associated with perceived system usefulness.
Technical performance of the developers, based on their
responsiveness to problems, was positively related to
improving efficiency (Leonard-Barton & Sinha, 1993). In
a case study on improving service quality, Blanton et al.
(1992) found that personalized IT support is more
effective than generalized IT support. However, the
developer skills for an expert system were not significantly
related to the impact on a user’s job (Yoon &
Guimaraes, 1995). Documentation of ERP systems was
also not significantly related to an individual’s perceived
performance (Kositanurit et al., 2006).
At the organizational unit of analysis more research
is clearly needed. Thong et al. (1994, 1996) found
that higher levels of vendor support and effectiveness
were related to lower operating costs. Gefen (2000)
determined that the greater the perception that the
vendor is cooperative, the greater the perceived usefulness
of a system.
Use-user satisfaction Surprisingly, little research has
examined the relationship between use and user
satisfaction. More research has examined the reverse
relationship, between user satisfaction and use, so
additional research is needed to evaluate this relationMeasuring
information systems success Stacie Petter et al246
European Journal of Information Systems
ship. One study, examining expert systems at DuPont,
found that system usage, measured as frequency of use,
and user satisfaction, measured using nine items from the
Bailey and Pearson instrument, were positively and
significantly related (Guimaraes et al., 1996). In a knowledge-management
context, Halawi et al. (2007) identified
a significant relationship between intention to use and
user satisfaction. Seddon & Kiew (1996) found that, in a
mandatory context, use, measured by system importance,
was not related to user satisfaction. Chiu et al. (2007)
identified a significant relationship between use and user
satisfaction in an e-learning context. However, Iivari
(2005) found, in a study of a medical information system
in which use was mandatory, that use measured by
amount of daily use and frequency of use was significantly
related to user satisfaction. While some researchers
have argued that use is irrelevant when a system is
mandatory, Iivari (2005) illustrates that it is possible to
have sufficient variability in the use construct to have
significant relationships with other constructs in the
D&M model, such as user satisfaction.
At the organizational level, Gelderman (1998), however,
found mixed results in significance in the correlations
between different measures of system use (i.e.,
frequency and duration) and user satisfaction.
Use-net benefits Empirical studies provide moderate
support for the relationship between system use and
benefits at the individual level. Several studies have
found that IS use is positively associated with improved
decision making. Yuthas & Young (1998) found that the
duration of system use is correlated with decision
performance. Burton-Jones & Straub (2006) found a
strongly significant relation between system usage and
task performance. Halawi et al. (2007) identified a
significant relationship between intention to use and
net benefits as measured by improvements in job
performance. Many studies confirm these findings by
finding significant relationships and/or correlations
between system use and net benefits (Goodhue &
Thompson, 1995; Yoon & Guimaraes, 1995; Seddon &
Kiew, 1996; Abdul-Gader, 1997; Guimaraes & Igbaria,
1997; Igbaria & Tan, 1997; Torkzadeh & Doll, 1999; Weill
& Vitale, 1999; D’Ambra & Rice, 2001; Rai et al., 2002;
Almutairi & Subramanian, 2005; Kositanurit et al., 2006).
On the other hand, some studies suggest otherwise.
One study found that intended use is not significantly
related to individual impact (task–technology fit and
performance) (McGill et al., 2003). Other studies also
found no relationship between use and net benefits
(Lucas & Spitler, 1999; Iivari, 2005; Wu & Wang, 2006).
Among users in three different Asian firms, there was no
significant relationship between frequency of use and job
satisfaction (Ang & Soh, 1997). In another series of
studies, the self-reported hours of use of IS among
managers was positively correlated with decision making
in a sample of German firms (Vlahos et al., 2004), but not
in Greek firms (Vlahos & Ferratt, 1995).
There is moderate support for the relationship between
system use and organizational benefits. Teng & Calhoun’s
study (1996) found that the intensity of IT usage had a
significant impact on job complexity, decision routinization,
and decision-making effectiveness. The results of a
study of IS use in a hospital setting confirmed a positive
relationship between system usage, as measured by the
number of DSS reports accessed and number of disk
accesses, and profitability and quality of care as measured
by decreased mortality (Devaraj & Kohli, 2003). Zhu &
Kraemer (2005) found that use of online IS for ebusinesses
had a positive, significant impact on value in
both developed and developing countries. Use of executive
IS did impact the productivity, decision-making, and
internal costs positively (Belcher & Watson, 1993).
Gelderman (1998) also found that system usage, in terms
of time duration, was not significantly correlated to
revenue and profitability improvement.
User satisfaction-use Studies examining the relationship
between user satisfaction and use have found
moderate support for this relationship at the individual
level (Iivari, 2005); however, the literature at the
organizational level of analysis is lacking. User satisfaction
is strongly related to use when measured by system
dependence (Rai et al., 2002; Kulkarni et al., 2006), the
frequency and duration of use (Guimaraes & Igbaria,
1997; Yuthas & Young, 1998), the number of applications
and tasks for which the information system is used
(Igbaria & Tan, 1997), and the intention to use (Kim et al.,
2002; McGill et al., 2003; Wu & Wang, 2006; Bharati &
Chaudhury, 2006; Chiu et al., 2007; Halawi et al., 2007).
Also, Wixom & Todd (2005) found a strong relationship
between satisfaction and intention to use when mediated
by technology acceptance constructs. Winter et al. (1998)
found that satisfaction with the system is correlated to
both the hours of use and the extensiveness of tasks in a
study of white collar workers.
Hsieh & Wang (2007) discovered a significant, positive
relationship between satisfaction and extent of use
among users of ERP systems in one of their research
models that examined the relationships among confirmation
of expectations, perceived usefulness, satisfaction,
and extent of use; however, this relationship
between satisfaction and extent of use became nonsignificant
when placed in a larger model that incorporated
perceived ease of use.
Several studies have determined the correlation between
user satisfaction and use. However, in these studies
we do not know the order of the relationship (i.e.,
whether use predicts user satisfaction or user satisfaction
predicts use). Several studies have found a significant
correlation between self-reported system usage and user
satisfaction (Abdul-Gader, 1997; Khalil & Elkordy, 1999;
Torkzadeh & Doll, 1999). Other studies, however, have
found conflicting results. For example, the self-reported
hours of use of IS was not significantly correlated to user
satisfaction in either German firms (Vlahos et al., 2004) or
Measuring information systems success Stacie Petter et al 247
European Journal of Information Systems
Greek firms (Vlahos & Ferratt, 1995). Frequency of use
was not significantly correlated with user satisfaction in
Asian firms (Ang & Soh, 1997). Collopy (1996) found that
actual usage of an information system was significantly
related to satisfaction; however, self-reported usage was
not significantly related to satisfaction.
User satisfaction-net benefits Empirical results have
shown a strong association between user satisfaction
and system benefits (Iivari, 2005). User satisfaction has
been found to have a positive impact on a user’s job
(Yoon & Guimaraes, 1995; Guimaraes & Igbaria, 1997;
Torkzadeh & Doll, 1999), to improve performance
(McGill et al., 2003), to increase productivity and
effectiveness (Igbaria & Tan, 1997; Rai et al., 2002; McGill
& Klobas, 2005; Halawi et al., 2007), to improve decision
making (Vlahos & Ferratt, 1995; Vlahos et al., 2004), and
to enhance job satisfaction (Ang & Soh, 1997; Morris
et al., 2002). However, Yuthas & Young (1998) found
that user satisfaction was only weakly correlated with
decision making performance. One study investigated
the association between user satisfaction and organizational
impact and found that satisfaction was correlated
to performance based on profitability and revenues
(Gelderman, 1998). Another study found similar results
when evaluating the relationship between user satisfaction
and organizational performance of ERP systems (Law
& Ngai, 2007).
Net benefits-use At the individual level of analysis, the
relationship between net benefits and use has received
moderate support. When measuring net benefits, using
perceived usefulness as the metric, many studies have
found a relationship between behavioral intention and
use of a system (Subramanian, 1994; Agarwal & Prasad,
1999; Venkatesh & Morris, 2000; Hong et al., 2001/2002;
Chau & Hu, 2002; Malhotra & Galletta, 2005; Wixom &
Todd, 2005; Klein, 2007). Other studies have found
strong relationships between perceived usefulness and
self-reported use (Straub et al., 1995; Igbaria et al., 1997;
Yang & Yoo, 2004; Wu & Wang, 2006) extent of use
(Hsieh & Wang, 2007) or dependence on an information
system (Kulkarni et al., 2006). Venkatesh et al. (2003)
found a significant relationship between performance
expectancy and intentions to use the system in both
voluntary and mandatory settings. These results were
consistent over time when measured one, three, and six
months after the implementation of the system. In a
study by Lucas & Spitler (1999) of brokers and sales
assistants at a financial institution, perceived usefulness
was not significantly related to intention to use, nor to
self-reported use of the system’s functionality. Straub
et al. (1995) also found no significant relationship
between perceived usefulness and actual use. Adams
et al. (1992) found both significant and non-significant
results in the relationship between perceived usefulness
and use, measured by frequency and duration, depending
on the information system under study. Another study
identified mixed results between benefits and use.
Compeau et al. (1999) found positive, significant results
when measuring net benefits as performance-related
outcomes, which is how the system affects one’s job;
however, when examining personal-related outcomes,
which is intrinsic feelings of accomplishment and the
perceptions by others, a small, negative significant effect
on computer use, measured by frequency and duration of
use, was identified. Agarwal & Prasad (1997) found no
relationship between the relative advantage of a system
in performing one’s work with self-reported frequency of
use; however, there was a significant relationship with
intention to use.
Examining this relationship from the organizational
level of analysis, studies have found strong support for
the relationship between net benefits and use. Perceived
usefulness is significantly related to self-reported use
(Gefen & Keil, 1998; Gefen, 2000). Belcher & Watson
(1993) performed an evaluation of executive information
systems (EIS) at a single organization using interviews to
assess performance and questionnaires to assess system
usage. The study found that the benefits of the EIS,
increased productivity of workers, improved decisionmaking
ability, and better information flow and connectivity
among employees, encouraged continued use of
the system. Premkumar et al. (1994) found a significant
relationship between the relative advantage of a system
measured by the increased profitability, sales, and
benefits of a system and the use of a system as measured
by initial use of the system and diffusion of use to
different types of activities within the firm; however,
complexity and compatibility with work processes
was not a significant predictor of initial use and diffusion
of use of the system. Gill (1996) identified factors such
as the type of knowledge required to complete one’s
work, the variety of tasks, and the identity developed
by using the system all had a significant, positive impact
on use.
Net benefits-user satisfaction There is strong support
for the relationship between net benefits and user
satisfaction. When Seddon & Kiew (1996) tested their
version of the IS success model, they opted to replace use
with perceived usefulness. Several studies (Seddon &
Kiew, 1996; Devaraj et al., 2002; Rai et al., 2002; Kulkarni
et al., 2006; Hsieh & Wang, 2007) have found a positive,
significant relationship between perceived usefulness
(i.e., net benefits) and user satisfaction. A qualitative
study also found a relationship between perceived
usefulness and user satisfaction (Leclercq, 2007). Three
other studies found that the impact an expert system has
on a user’s job directly affects user satisfaction (Yoon
et al., 1995; Guimaraes et al., 1996; Wu & Wang, 2006).
These studies, examining the individual unit of analysis,
provide support for this relationship. Abdul-Gader (1997)
found a significant correlation between perceived productivity
and user satisfaction of computer-mediated
communication systems in Saudi Arabia. A relationship
Measuring information systems success Stacie Petter et al248
European Journal of Information Systems
between decision-making satisfaction and overall user
satisfaction was also discovered in a study of e-commerce
Web sites (Bharati & Chaudhury, 2006).
At the organizational unit of analysis, however, there is
not enough data to comment on the relationship
between net benefits and user satisfaction. Two studies
examining different aspects of net benefits have yielded
similar results. Net benefits, as measured by organizational
benefits or impacts, was significantly related to
user satisfaction (Teo & Wong, 1998; Jones & Beatty,
2001). Yet, in another study, an IS’ effect on the work
environment was not significantly related to user satisfaction
(Teo & Wong, 1998; Jones & Beatty, 2001).
Premkumar et al. (1994) found no relationship between
relative advantage (increased profitability, sales, and
benefits of a system) and user satisfaction; however,
compatibility with an individual’s work practices did
significantly predict user satisfaction.
Implications and conclusions
Summary of the results
Empirical support for the relationships between the
success dimensions are summarized in Tables 3 and 4
and Figures 3 and 4. We classified the level of support for
each relationship as strong, moderate, or mixed in order
to summarize the empirical results across all studies. To
classify the level of support for each relationship, we
assigned studies with significant, positive results 1.0
point; studies with both significant, positive results and
non-significant results (i.e., mixed results) 0.5 points;
and studies with a non-significant relationship between
the constructs 0.0 points. The sum of the points was
then divided by the number of studies. To assign values
of strong, moderate, or mixed, we examined the
distribution of percentages. ‘Strong’ support was assigned
when the percentage of papers with a positive result
was in the range of 90–100%; ‘moderate’ support for a
range of 67–83%, and ‘mixed’ support with a range of
25–53%. These percentages are actually quite conservative
in that if we used a more stringent quantitative
approach, such as vote counting, many of the relationships
labeled as moderate and mixed support would
show that a significant relationship exists between
the constructs (Hedges & Olkin, 1980). However, the
purpose of our literature review is not to reduce the
relationship between constructs to a number (or effect
size), but rather to suggest areas that deserve additional
research. To err on the side of caution, any relationship
that had four or less studies examining the relationship
was deemed as having insufficient data to draw a
conclusion.
Table 3 Summary of empirical studies at an individual level of analysis
Relationship Empirical studies Study result Overall result Conclusion
System quality-use Halawi et al. (2007) + 9 of 21 found a positive association Mixed support
Hsieh & Wang (2007) +
Iivari ( 2005) +
Rai et al. (2002) +
Hong et al. (2001/2002) +
Venkatesh & Davis (2000) +
Venkatesh & Morris (2000) +
Igbaria et al. (1997) +
Suh et al. (1994)*a
+
Kositanurit et al. (2006) Mixed
Venkatesh et al. (2003) Mixed
Agarwal & Prasad (1997) Mixed
Goodhue & Thompson (1995) Mixed
Adams et al. (1992) Mixed
Klein (2007) NS
McGill et al. (2003) NS
Lucas & & Spitler (1999) NS
Gefen & Keil (1998) NS
Straub et al. (1995) NS
Markus & Keil (1994) NS
Subramanian (1994) NS
System quality-user satisfaction Chiu et al. (2007) + 21 of 21 positive Strong support
Halawi et al. (2007) +
Hsieh & Wang (2007) +
Leclercq (2007) +
Kulkarni et al. (2006) +
Wu & Wang (2006) +
Measuring information systems success Stacie Petter et al 249
European Journal of Information Systems
Table 3 Continued
Relationship Empirical studies Study result Overall result Conclusion
Almutairi & Subramanian (2005) +
Iivari (2005) +
McGill & Klobas (2005) +
Wixom & Todd (2005) +
McGill et al. (2003) +
Bharati (2002) +
Devaraj et al. (2002) +
Gelderman (2002) +
Kim et al. (2002) +
Palmer (2002) +
Rai et al. (2002) +
Guimaraes et al. (1996)* +
Seddon & Kiew (1996) +
Yoon et al. (1995) +
Seddon & Yip (1992) +
System quality-net benefits Hsieh & Wang (2007) + 15 of 22 positive Moderate support
Klein (2007) +
Bharati & Chaudhury, 2006 +
Wixom & Todd (2005) +
Shih (2004) +
Yang & Yoo (2004) +
Rai et al. (2002) +
Devaraj et al. (2002) +
Hong et al. (2001/2002) +
Venkatesh & Davis (2000) +
Venkatesh & Morris (2000) +
Agarwal & Prasad (1999) +
Lucas & Spitler (1999) +
Gefen & Keil (1998) +
Seddon & Kiew (1996) +
Kositanurit et al. (2006) Mixed
Kulkarni et al. (2006) NS
Wu & Wang (2006) NS
McGill & Klobas (2005) NS
Chau & Hu (2002) NS
Goodhue & Thompson (1995) NS
Subramanian (1994) NS
Information quality-use Halawi et al. (2007) + 3 of 6 positive Mixed support
Kositanurit et al. (2006) +
Rai et al. (2002) +
Goodhue & Thompson (1995) Mixed
McGill et al. (2003) NS
Iivari (2005) NS
Information quality-user satisfaction Chiu et al. (2007) + 15 of 16 positive Strong support
Halawi et al. (2007) +
Leclercq (2007) +
Kulkarni et al. (2006) +
Wu & Wang (2006) +
Almutairi & Subramanian (2005) +
Iivari (2005) +
Wixom & Todd (2005) +
McGill et al. (2003) +
Bharati (2002) +
Kim et al. (2002) +
Palmer (2002) +
Rai et al. (2002) +
Measuring information systems success Stacie Petter et al250
European Journal of Information Systems
Table 3 Continued
Relationship Empirical studies Study result Overall result Conclusion
Seddon & Kiew (1996) +
Seddon & Yip (1992) +
Marble (2003) NS
Information quality-net benefits Bharati & Chaudhury, 2006 + 9 of 11 positive Moderate support
Kositanurit et al. (2006) +
Wu & Wang (2006) +
Shih (2004) +
Rai et al. (2002) +
D’Ambra & Rice (2001) +
Seddon & Kiew (1996) +
Gatian (1994) +
Kraemer et al. (1993) +
Hong et al. (2001/2002) Mixed
Kulkarni et al. (2006) NS
Service quality-use Choe (1996) Mixed 0 of 3 positive Insufficient data
Halawi et al. (2007) NS
Kositanurit et al. (2006) NS
Service quality-user satisfaction Halawi et al. (2007) + 6 of 12 positive Mixed support
Leclercq (2007) +
Shaw et al. (2002) +
Yoon et al. (1995) +
Kettinger & Lee (1994) +
Leonard-Barton & Sinha (1993) +
Devaraj et al. (2002) Mixed
Chiu et al. (2007) NS
Marble (2003) NS
Aladwani (2002) NS
Palmer (2002) NS
Choe (1996) NS
Service quality-net benefits Agarwal & Prasad (1999) + 4 of 7 positive Moderate support
Gefen & Keil (1998) +
Leonard-Barton & Sinha (1993) +
Blanton et al. (1992) +
Igbaria et al. (1997) Mixed
Kositanurit et al. (2006) NS
Yoon & Guimaraes (1995)* NS
Use-user satisfaction Chiu et al. (2007) + 4 of 5 positive Moderate support
Halawi et al. (2007) +
Iivari (2005) +
Guimaraes et al. (1996) +
Seddon & Kiew (1996) NS
Use-net benefits Halawi et al. (2007) + 16 of 22 positive Moderate support
Burton-Jones & Straub (2006) +
Kositanurit et al. (2006) +
Almutairi & Subramanian (2005) +
Vlahos et al. (2004)* +
Rai et al. (2002) +
D’Ambra & Rice (2001) +
Torkzadeh & Doll (1999)* +
Weill & Vitale (1999) +
Yuthas & Young (1998)* +
Abdul-Gader (1997)* +
Guimaraes & Igbaria (1997) +
Measuring information systems success Stacie Petter et al 251
European Journal of Information Systems
Table 3 Continued
Relationship Empirical studies Study result Overall result Conclusion
Igbaria & Tan (1997) +
Seddon & Kiew (1996) +
Goodhue & Thompson (1995) +
Yoon & Guimaraes (1995)* +
Wu & Wang (2006) NS
Iivari (2005) NS
McGill et al. (2003) NS
Lucas & Spitler (1999) NS
Ang & Soh (1997)* NS
Vlahos & Ferratt (1995)* NS
User satisfaction-use Chiu et al. (2007) + 17 of 21 positive Moderate support
Halawi et al. (2007) +
Bharati & Chaudhury, 2006 +
Kulkarni et al. (2006) +
Wu & Wang (2006) +
Iivari (2005) +
Wixom & Todd (2005) +
McGill et al. (2003) +
Kim et al. (2002) +
Rai et al. (2002) +
Torkzadeh & Doll (1999)* +
Khalil & Elkordy (1999)* +
Winter et al. (1998)* +
Yuthas & Young (1998)* +
Abdul-Gader (1997)* +
Guimaraes & Igbaria (1997) +
Igbaria & Tan (1997) +
Collopy (1996) Mixed
Vlahos et al. (2004)* NS
Ang & Soh (1997)* NS
Vlahos & Ferratt (1995)* NS
User satisfaction-net benefits Halawi et al. (2007) + 14 of 14 positive Strong support
Iivari (2005) +
McGill & Klobas (2005) +
Vlahos et al. (2004)* +
McGill et al. (2003) +
Morris et al. (2002) +
Rai et al. (2002) +
Torkzadeh & Doll (1999)* +
Yuthas & Young (1998)* +
Ang & Soh (1997)* +
Guimaraes & Igbaria (1997) +
Igbaria & Tan (1997) +
Vlahos & Ferratt (1995)* +
Yoon & Guimaraes (1995)* +
Net benefits-use Hsieh & Wang (2007) + 15 of 21 positive Moderate support
Klein (2007) +
Wu & Wang (2006) +
Malhotra & Galletta (2005) +
Wixom & Todd (2005) +
Yang & Yoo (2004) +
Venkatesh et al. (2003) +
Chau & Hu (2002) +
Rai et al. (2002) +
Hong et al. (2001/2002) +
Venkatesh & Morris (2000) +
Measuring information systems success Stacie Petter et al252
European Journal of Information Systems
Table 3 Continued
Relationship Empirical studies Study result Overall result Conclusion
Agarwal & Prasad (1999) +
Gefen & Keil (1998) +
Igbaria et al. (1997) +
Subramanian (1994) +
Compeau et al. (1999) Mixed
Agarwal & Prasad (1997) Mixed
Straub et al. (1995) Mixed
Adams et al. (1992) Mixed
Kulkarni et al. (2006) NS
Lucas & Spitler (1999) NS
Net benefits-user satisfaction Hsieh & Wang (2007) + 11 of 11 positive Strong support
Leclercq (2007) +
Bharati & Chaudhury, 2006 +
Kulkarni et al. (2006) +
Wu & Wang (2006) +
Devaraj et al. (2002) +
Rai et al. (2002) +
Abdul-Gader (1997) +
Guimaraes et al. (1996) +
Seddon & Kiew (1996) +
Yoon et al. (1995) +
a
Studies listed in Tables 3 and 4 with an asterisk are studies that have found a correlational association between the constructs, rather than a causal
relationship.
Table 4 Summary of empirical studies at an organizational level of analysis
Relationship Empirical studies Study result Overall result Conclusion
System quality-use Fitzgerald & Russo (2005) + 2 of 6 found a positive association Mixed support
Caldeira & Ward (2002) +
Weill & Vitale (1999) À
Premkumar et al. (1994) Mixed
Gefen (2000) NS
Gill (1995) NS
System quality-user satisfaction Scheepers et al. (2006) + 2 of 3 Insufficient data
Benard & Satir (1993) +
Premkumar et al. (1994) NS
System quality-net benefits Wixom & Watson (2001) + 4 of 5 positive Moderate support
Gefen (2000) +
Weill & Vitale (1999) +
Farhoomand & Drury (1996) +
Bradley et al. (2006) Mixed
Information quality-use Fitzgerald & Russo (2005) + 1 of 1 positive Insufficient data
Information quality-user satisfaction Scheepers et al. (2006) + 3 of 3 positive Insufficient data
Coombs et al. (2001) +
Teo & Wong (1998) +
Information quality-net benefits Wixom & Watson (2001) + 3 of 4 positive Insufficient data
Teo & Wong (1998) +
Farhoomand & Drury (1996) +
Bradley et al. (2006) Mixed
Measuring information systems success Stacie Petter et al 253
European Journal of Information Systems
Table 3 and Figure 3 show the summary of the
literature review at an individual level of analysis, while
Table 4 and Figure 4 summarizes the results at an
organizational level of analysis. The tables and figures
clearly indicate that there is a paucity of research
examining IS success at the organizational level of
analysis. Most of the research relies on individual
subjects reporting on their own perceptions about an
information system. D&M have claimed that their
updated model could be used at any level of analysis.
However, more work is needed to confirm whether this is
indeed true.
Visual comparison of Figures 3 and 4 shows that,
although there are insufficient studies at the organization
level to evaluate the strength of most of the relationships,
there is consistent support across units of analysis for the
following relationships: system quality –4 net benefits
and net benefits –4use. High-quality systems lead to
greater net benefits. Systems that yield higher net
benefits are used to a greater degree.
One limitation of any literature review (either a
qualitative literature review or a quantitative metaanalysis)
is that the findings are highly dependent on
the literature identified, examined, and analyzed as part
of the review. In an effort to determine whether our
results are consistent with other literature reviews, we
compared our results to Sabherwal et al.’s (2006) metaanalyses
examining IS success. Table 5 compares the
results of the literature review (at the individual level of
analysis) to the results of Sabherwal et al.’s study.
The relationships between system quality and use, user
satisfaction, and net benefits all had varying levels of
support across both studies. This study also extends the
prior work performed by Sabherwal et al. by evaluating
the significant relationships among success dimensions
that were not examined in their meta-analysis. RelationTable
4 Continued
Relationship Empirical studies Study result Overall result Conclusion
Service quality-use Fitzgerald & Russo (2005) + 3 of 3 positive Insufficient data
Caldeira & Ward (2002) +
Gill (1995) +
Service quality-user satisfaction Coombs et al. (2001) + 3 of 4 positive Insufficient data
Thong et al. (1996) +
Thong et al. (1994) +
Benard & Satir (1993) NS
Service quality-net benefits Gefen (2000) + 3 of 3 positive Insufficient data
Thong et al. (1996) +
Thong et al. (1994) +
Use-user satisfaction Gelderman (1998)* Mixed 0 of 1 positive Insufficient data
Use-net benefits Leclercq (2007) + 5 of 6 positive Moderate support
Zhu & Kraemer (2005) +
Devaraj & Kohli (2003) +
Teng & Calhoun (1996) +
Belcher & Watson (1993) +
Gelderman (1998)* NS
User satisfaction-use No studies Insufficient data
User satisfaction-net benefits Gelderman (1998)* + 2 of 2 positive Insufficient data
Law & Ngai (2007) +
Net benefits-use Gefen (2000) + 3 of 4 positive Insufficient data
Gill (1996) +
Belcher & Watson (1993) +
Premkumar et al. (1994) Mixed
Net benefits-user satisfaction Jones & Beatty (2001) Mixed 0 of 3 positive Insufficient data
Teo & Wong (1998) Mixed
Premkumar et al. (1994) Mixed
Measuring information systems success Stacie Petter et al254
European Journal of Information Systems
ships associated with information quality and service
quality were not examined in Sabherwal et al. but were
examined here which supports the proposed relationship
between information quality and user satisfaction and
between information quality and net benefits. Also, the
relationships between net benefits with use and user
satisfaction were supported.
The literature on user satisfaction and use has had
conflicting results. Our literature review offers moderate
support, yet Sabherwal et al. found no support in their
structural equation model when examining this relationship.
This difference across studies could be due to the
different measures that were used across studies for both
use and user satisfaction. Probably the most striking
finding is that we found strong support for the relationship
between net benefits and user satisfaction, while
Sabherwal et al. had a non-significant path for this
relationship. One difference between these studies is that
Sabherwal et al. focused their assessment of net benefits
on studies that measured perceived usefulness. Our
literature review included studies that used a broader
set of measures for net benefits in addition to perceived
usefulness. Another value of qualitative research compared
to meta-analyses is the ability to understand the
direction of association among variables. The empirical
studies included in this literature review confirm the
System
Quality
Information
Quality
Service
Quality
Use
User
Satisfaction
Net Benefits
Moderate to Strong Support
Mixed Support
Insufficient Data
Figure 3 Support for interrelationships between D&M success
constructs at an individual level of analysis.
System
Quality
Information
Quality
Service
Quality
Use
User
Satisfaction
Net Benefits
Moderate to Strong Support
Mixed Support
Insufficient Data
Figure 4 Support for interrelationships between D&M success
constructs at an organizational level of analysis.
Table 5 Comparison of literature review results to meta-analyses results
Relationship Literature review (Individual) Sabherwal et al. (2006)a
System quality-use Mixed support Significant
System quality-user satisfaction Strong support Significant
System quality-net benefits Moderate support Significant
Information quality-use Insufficient data Not examined
Information quality-user Satisfaction Strong support Not examined
Information quality-net benefits Moderate support Not examined
Service quality-use Insufficient data Not examined
Service quality-user satisfaction Mixed support Not examined
Service quality-net benefits Moderate support Not examined
Use-user satisfaction Insufficient data Not examined
Use-net benefits Moderate support Significant (correlation)
User satisfaction-use Moderate support Not significant
User satisfaction-net benefits Strong support Not examined
Net benefits-use Moderate support Significant (correlation)
Net benefits-user satisfaction Strong support Not significant
a
The results reported here are based on the structural equation modeling assessment that Sabherwal et al. performed on their research model after
conducting a series of meta-analyses.
Measuring information systems success Stacie Petter et al 255
European Journal of Information Systems
direction of the relationships proposed in the D&M IS
success model. For example, we find that higher levels of
user satisfaction result in greater use but that there is
insufficient empirical evidence to know whether more
use will lead to greater user satisfaction. Importantly, the
literature review also reveals that increased net benefits
lead to a higher degree systems use and to a higher level
of user satisfaction.
Although a qualitative literature review is subject to the
interpretation of its authors, the overall assessments of IS
success measurement formed as a result of this review are
consistent with, and extend beyond, the results of the
quantitative methods that have been employed pre-
viously.
Implications for researchers
Measuring success
Although many research studies have tested and validated
IS success measurement instruments, most have
focused on a single dimension of success such as system
quality, benefits, or user satisfaction. Few studies measure
and account for the multiple dimensions of success and
the interrelationships among these dimensions. Until IS
empirical studies consistently apply a validated, multidimensional
success measure, the IS field will be plagued
with inconsistent results and an inability to generalize its
findings. Given that many studies only capture one
dimension of each success construct (i.e., perceived ease
of use for system quality), measures like those applied in
Sedera et al.’s (2004) multidimensional success measurement
instrument provide higher content validity. Their
research has proven to be a valid and reliable step toward
improved IS success measurement and either their
instrument or their approach for creating and validating
instruments should be adopted and further tested in
different contexts.
Another problem for IS success measurement is the
proclivity to use only the user satisfaction dimension as a
surrogate measure of success. Studies have found that
self-reported measures are not consistent with actual
measures (Heo & Han, 2003). Subjective measures are
therefore not always a very reliable substitute for
objective measures of success. Nevertheless, many empirical
studies of IS effectiveness adopt user satisfaction as
a primary measure or surrogate of IS success due partially
to convenience and ease of measurement.
One finding in this research is that the various
measures of success used in the studies examined for
the literature review could be classified using one or more
of the dimensions of the D&M IS success model. This is
clearly a strength of the D&M model; however, several
studies could not be classified in the literature review
because of the use of a general effectiveness measure that
measures multiple dimensions of success (Sedera et al.,
2004). The utilization of user satisfaction as a surrogate
for success masks the important dimensions of success
such as system quality, information quality, and net
benefits. Generally, most researchers do not take the time
to parse the different dimensions from the user satisfaction
instrument to measure specific dimensions of
success. User satisfaction in and of itself is often a goal
for IS and therefore worthy of measurement, but
satisfaction should not be used alone as the sole indicator
of success. This is part of the reason for the lack of studies
examining the relationship between information quality
and other success constructs.
The results of the literature review reported here
emphasize the current problems the field is experiencing
with measuring and understanding system use. The
relationship between each dimension and use is lower
than associations with any other dimension; that is,
system quality and use has less support than system
quality and user satisfaction and system quality and net
benefits. The inconsistency regarding the use construct
seems to be largely related to the measurement and
understanding of the use construct. System use is a
success construct that is often criticized and/or ignored.
Its significance has found mixed results in empirical
studies. It is our contention that system use is an
important indicator of IS success and associated with
the ultimate impact or benefits garnered from IS. Much
like the measurement of success as a whole, we believe
that the measure of system use has been over simplified;
ignored when use is mandatory and poorly measured as
merely frequency or time of use when voluntary.
Use is not a dichotomous variable on the volition scale;
that is, it is rarely ever either totally voluntary or totally
mandatory. A three-year case study of eight organizations
found that IS use within the organization was not simply
a question of use or non-use, but rather that the
utilization was complex, with transition and equilibrium
periods (Lassila & Brancheau, 1999). Also, perceived use is
not a very satisfactory method for measuring use.
Intensity, purpose, sophistication, extents, and more
should be included in the study of use. Researchers
should always include use as a measure of success and
omit it only after empirical results demonstrate that it
provides little or no added explanatory value beyond the
other dimensions of success in the study under con-
sideration.
A final issue associated with the measurement of
constructs is the dependence on measures from TAM.
First, content validity suffers because system quality is
only measured in terms of ease of use and net benefits is
only measured in terms of perceived usefulness. In the
studies examining the relationship between system
quality and use, there was mixed support for this
relationship if the researcher used only perceived ease
of use as only the measure of system quality. Measuring
system quality using measures other than, or in addition
to, perceived ease of use more consistently predicted the
different measures of use.
More importantly, this focus on TAM keeps researchers
focused at the individual level of analysis. Since most
studies have focused on the benefits to an individual, we
Measuring information systems success Stacie Petter et al256
European Journal of Information Systems
have little understanding of the impact of an information
system at an organizational level as found and described
in this study. Researchers need to make a conscious effort
to consider studying the concept of net benefits beyond
the individual and consider group, organizational, and
even possibly societal impacts.
Understanding the relationships
By separating the studies examining the individual
context from those examining the organizational context,
some interesting findings come to light. First, it was
discovered that there are few studies that have considered
the relationships that comprise the D&M model from an
organizational point of view. Only three of the 15
relationships among the success dimensions have received
any reasonable level of study. It was also clear that
by grouping studies in terms of their context (i.e.,
individual or organizational), it was possible to better
understand some of the relationships. For example, if the
studies are aggregated across contexts for system quality
and benefits, there is moderate support. However, this
grouping of studies reveals that there is stronger support
for this relationship at the organizational level than at
the individual level. This same phenomenon occurs for
the relationship between benefits and use. The organizational
studies of this relationship have more consistently
found a relationship between these constructs than
studies at the individual level. If, for example, the studies
for information quality and use are aggregated, the result
is moderate support; however, careful parsing of these
studies by context shows that there are really insufficient
data to evaluate these relationships at the individual and
organizational levels.
This qualitative literature review did not restrict studies
by the mandatory or voluntary nature of the system use,
or by the type of information system examined. It may be
that relationships with strong support are not subject to
boundary conditions such as the voluntariness of the
system or the type of information system; however, more
research is needed to confirm this conjecture.
Studies have shown that service quality is associated
with individual performance; therefore, service quality
deserves to be included as a dimension of IS success.
Despite some controversy, SERVQUAL has been shown to
be a valid measure of IS service quality. Failure to consider
system quality, information quality, and/or service quality
can lead to confounding results where the implied
negative impact of an independent variable may actually
be the result of poor systems-related dimensions that
have not been taken into account.
Finally, information quality is a neglected aspect of IS
success. Since a primary motivation for IS applications is
to supply managers with accurate, timely, and relevant
information, information quality is an important dimension
of a system’s success. Information quality is also an
important factor in measuring user satisfaction and
should be treated as a success measure separate from
the popular end-user satisfaction instrument (Doll et al.,
1994). Since information relevance is an important
dimension of information quality and can vary widely
by systems, there is likely to be high variance in
information quality in practice; therefore, this variance
should be accounted for in empirical IS research (Sedera
et al., 2004).
Implications for practice
Practitioners consistently acknowledge the importance of
measuring the value of their IS investments. However,
practitioner IS-effectiveness measurement methods suffer
the same deficiency as academic success models; that is,
they are often one-dimensional and over-simplified.
Practitioners tend to focus on net impacts or benefits
but fail to consider system, information, and service
quality as well as the nature and intensity of system use.
As for the important net benefits dimension, Kaplan
and Norton’s ‘Balanced Scorecard’ (1996) holds promise
for measuring the business contribution of IS. Martinsons
et al. (1999) proposed a balanced IS scorecard that
consists of four performance dimensions: namely, a
business-value dimension, a user-oriented dimension,
an internal-process dimension, and a future-readiness
dimension.
According to a study by Heo & Han (2003), the
constructs of the IS success model have different degrees
of importance depending on the firm characteristics.
Firms with more centralized computing place emphasis
on performance measures in the following order: system
quality, information quality, user satisfaction, use, organizational
impact, and individual impact. Firms with
decentralized computing environments emphasize information
quality and system quality highly; emphasize
user satisfaction, individual impact, and organizational
impact moderately; and information use as the least
important measure. For firms with centralized cooperative
computing, organizational impact and system quality
were the most important measures, followed by
information quality, user satisfaction, individual impact,
and use. For firms with distributed cooperative computing,
organizational impact is the most important factor as
well as individual impact and information quality. Use
was found to be the least appropriate measure for this
group.
In another study of the importance of various IS success
factors to managers, Li (1997) found that accuracy of
output, reliability of output, relationship between users
and the CBIS staff, user’s confidence in the systems, and
timeliness of output were the most important factors. The
five least important factors were the chargeback method,
volume of output, competition between CBIS and nonCBIS
units, features of the computer language used, and
job effects of computer-based support.
Rainer & Watson (1995) studied factors associated with
the success of executive IS, using the D&M model, and
found ease of use, information quality, and impact on
work were critical factors in these types of systems.
Measuring information systems success Stacie Petter et al 257
European Journal of Information Systems
Practitioners are advised to deploy success measurement
programs that incorporate all six dimensions of IS
success: system quality, information quality, service
quality, objective (as opposed to subjective) measures of
system use, user satisfaction, and net benefits. The
context, purpose, unit of analysis (individual vs organizational),
and importance of systems should dictate the
relative weights to place on each of these success
dimensions and measures. An IS balanced scorecard
should also be considered as a way of measuring net
benefits.
Future research
While recent research has provided strong support for
many of the proposed interrelationships among success
dimensions in the D&M model, more research is needed
to explore the relationships that have not been adequately
researched. Empirical research is also needed to
establish the strength of interrelationships across different
contextual boundaries. This study takes a first step by
parsing out the results based on individual vs organizational
units of analysis and found that there is insufficient
empirical evidence to evaluate most of the
relationships at the organizational level. However, there
could be other, more complex effects that could explain
the relationship between these success constructs at
either the individual or organizational levels of analysis.
Researchers may want to consider complex functions,
such as curvilinear effects, that affect the relationships
among IS success constructs.1
There are also a number of
other boundary conditions that deserve attention, such
as the voluntariness of the system, the timing of success
measurement (i.e., the difference between the time of the
implementation of the system and the time of measurement),
and the type of information system examined.
Secondly, more research is needed on the relationships
between information quality and use, user satisfaction,
and net benefits. Finally, IS researchers still struggle with
system use as a measure of IS success. Future studies must
apply more comprehensive and consistent measures of
use in order to better understand the effect of use on user
satisfaction and net benefits. Burton-Jones & Straub
(2006) have taken an important step in improving the
measurement of systems use by incorporating the
structure and function of use.
Conclusion
This literature review examined IS success at both
individual and organizational levels of analysis. The
D&M IS success model applied equally well at both the
individual and organizational levels of analysis in those
cases where there were sufficient data to analyze the
relationships. This research also considered many different
types of IS under a variety of conditions and had
reasonable support for the majority of relationships
within the model, suggesting the value of the D&M
model of IS success when evaluating utilitarian IS. What
still remains to be discovered is if the D&M model is
appropriate for hedonic IS. Some of the dimensions may
no longer be relevant or may need to be measured
differently for gaming, social networking, or other types
of IS used for enjoyment.
The science of measuring information success or
performance in empirical studies has seen little improvement
over the past decade. Researchers and practitioners
still tend to focus on single dimensions of IS success and
therefore do not get a clear picture of the impacts of their
systems and methods. Progress in measuring the individual
success dimensions has also been slow. The work of
Sedera et al. (2004) in developing measures for success is
encouraging and this type of work should be continued
in future research. Valid and reliable measures have yet to
be developed and consistently applied for system quality,
information quality, use, and net benefits. The D&M IS
success model (2003) is a useful framework for understanding
the key success dimensions and their interrelationships.
However, researchers must take a step further
and apply rigorous success measurement methods to
create comprehensive, replicable, and informative measures
of IS success.
About the authors
Stacie Petter is an assistant professor at the University of
Nebraska at Omaha. She has published her research in the
MIS Quarterly, Information Sciences, and IT Professional. Her
research examines topics related to software project
management, knowledge management, information systems
success, and research methods.
William DeLone is a professor of Information Systems
and Director of the Center for Information Technology
and the Global Economy at the Kogod School of Business
at American University in Washington, DC. His research
interests include the assessment of information systems
success and the management of global software development.
He has published in Information Systems Research,
MIS Quarterly, Journal of MIS, and the Communications of
the ACM.
Ephraim R. McLean is a Regents’ professor and holder
of the Smith Eminent Scholar’s Chair in Information
Systems in the Robinson College of Business at Georgia
State University, in Atlanta, GA where he is also the
Chairman of the CIS Department. He is a fellow of
AIS and in 2007 was recognized with the LEO
Lifetime Achievement Award from AIS and ICIS. His
research focuses on the management of information
technology.1
We thank an anonymous reviewer for suggesting this point.
Measuring information systems success Stacie Petter et al258
European Journal of Information Systems
References
ABDUL-GADER AH (1997) Determinants of computer-mediated communication
success among knowledge workers in Saudi Arabia. Journal of
Computer Information Systems 38(1), 55–66.
ADAMS DA, NELSON RR and TODD PA (1992) Perceived usefulness, ease of
use, and usage of information technology: a replication. MIS Quarterly
16(2), 227–247.
AGARWAL R and PRASAD J (1997) The role of innovation characteristics and
perceived voluntariness in the acceptance of information technologies.
Decision Sciences 28(3), 557–582.
AGARWAL R and PRASAD J (1999) Are individual differences germane to the
acceptance of new information technologies?. Decision Sciences 30(2),
361–391.
ALADWANI AM (2002) Organizational actions, computer attitudes, and
end-user satisfaction in public organizations: an empirical study.
Journal of End User Computing 14(1), 42–49.
ALMUTAIRI H and SUBRAMANIAN GH (2005) An empirical application of the
DeLone and McLean model in the Kuwaiti private sector. Journal of
Computer Information Systems 45(3), 113–122.
ANG S and SOH C (1997) User information satisfaction, job satisfaction,
and computer background: an exploratory study. Information &
Management 32(5), 255–266.
AU N, NGAI EWT and CHENG TCE (2002) A critical review of end-user
information system satisfaction research and a new research framework.
Omega 30(6), 451–478.
BALLANTINE J, BONNER M, LEVY M, MARTIN AIM and POWELL PL (1996) The
3-D model of information systems success: the search for the
dependent variable continues. Information Resources Management
Journal 9(4), 5–14.
BAROUDI JJ and ORLIKOWSKI WJ (1988) A short-form measure of user
information satisfaction: a psychometric evaluation and notes on use.
Journal of Management Information Systems 4(4), 44–59.
BELCHER LW and WATSON HJ (1993) Assessing the value of Conoco’s EIS.
MIS Quarterly 17(3), 239–254.
BENARD R and SATIR A (1993) User satisfaction with EISs – meeting
the needs of executive users. Information Systems Management 10(4),
21–29.
BHARATI P (2002) People and information matter: task support satisfaction
from the other side. Journal of Computer Information Systems 43(2),
93–102.
BHARATI P and CHAUDHURY A (2006) Product customization on the web: an
empirical study of factors impacting choiceboard user satisfaction.
Information Resources Management Journal 19(2), 69–81.
BLANTON JE, WATSON HJ and MOODY J (1992) Toward a better
understanding of information technology organization: a comparative
case study. MIS Quarterly 16(4), 531–555.
BOKHARI RH (2005) The relationship between system usage and user
satisfaction: a meta-analysis. The Journal of Enterprise Information
Management 18(2), 211–234.
BRADLEY RV, PRIDMORE JL and BYRD TA (2006) Information systems success
in the context of different corporate culture types: an empirical
investigation. Journal of Management Information Systems 23(2),
267–294.
BRYNJOLFSSON E, HITT LM and YANG S (2002) Intangible assets: how
computers and organizational structure affect stock market valuations.
Brookings Papers on Economic Activity 1, 137.
BURTON-JONES A and GALLIVAN MJ (2007) Toward a deeper understanding
of system usage in organizations: a multilevel perspective. MIS
Quarterly 31(4), 657–680.
BURTON-JONES A and STRAUB D (2006) Reconceptualizing system usage:
an approach and empirical test. Information Systems Research 17(3),
220–246.
CALDEIRA MM and WARD JM (2002) Understanding the successful
adoption and use of IS/IT in SMEs: an explanation from Portuguese
manufacturing industries. Information Systems Journal 12(2), 121–152.
CHAU PYK and HU PJ (2002) Examining a model of information
technology acceptance by individual professionals: an exploratory
study. Journal of Management Information Systems 18(4), 191–230.
CHIU CM, CHIU CS and CHANG HC (2007) Examining the integrated
influence of fairness and quality on learners’ satisfaction and Webbased
learning continuance intention. Information Systems Journal
17(3), 271–287.
CHOE JM (1996) The relationships among performance of accounting
information systems, influence factors, and evolution level of information
systems. Journal of Management Information Systems 12(4),
215–239.
COLLOPY F (1996) Biases in retrospective self-reports on time use:
an empirical study of computer users. Management Science 42(5),
758–767.
COMPEAU D, HIGGINS CA and HUFF S (1999) Social cognitive theory and
individual reactions to computing technology: a longitudinal study.
MIS Quarterly 23(2), 145–158.
COOMBS CR, DOHERTY NF and LOAN-CLARKE J (2001) The importance of
user ownership and positive user attitudes in the successful adoption
of community information systems. Journal of End User Computing
13(4), 5–16.
D’AMBRA J and RICE RE (2001) Emerging factors in user evaluation of the
World Wide Web. Information & Management 38(6), 373–384.
DAVIS FD (1989) Perceived usefulness, perceived ease of use, and
user acceptance of information technology. MIS Quarterly 13(3),
318–346.
DELONE WH and MCLEAN ER (1992) Information systems success: the quest
for the dependent variable. Information Systems Research 3(1), 60–95.
DELONE WH and MCLEAN ER (2002) Information systems success revisited.
In Proceedings of the 35th Hawaii International Conference on System
Sciences (SPRAGUE JR RH, Ed) p 238, IEEE Computer Society, Hawaii, US.
DELONE WH and MCLEAN ER (2003) The DeLone and McLean model of
information systems success: a ten-year update. Journal of Management
Information Systems 19(4), 9–30.
DELONE WH and MCLEAN ER (2004) Measuring e-commerce success:
applying the DeLone & McLean information systems success model.
International Journal of Electronic Commerce 9(1), 31–47.
DEVARAJ S, FAN M and KOHLI R (2002) Antecedents of B2C channel
satisfaction and preference: validating e-commerce metrics. Information
Systems Research 13(3), 316–333.
DEVARAJ S and KOHLI R (2003) Performance impacts of information
technology: is actual usage the missing link?. Management Science
49(3), 273–289.
DOLL WJ and TORKZADEH G (1998) Developing a multidimensional
measure of system-use in an organizational context. Information &
Management 33(4), 171–185.
DOLL WJ, XIA W and TORKZADEH G (1994) A confirmatory factor analysis of
the end-user computing satisfaction instrument. MIS Quarterly 18(4),
453–461.
FARHOOMAND AF and DRURY DH (1996) Factors influencing electronic data
interchange success. The DATA BASE for Advances in Information
Systems 27(1), 45–57.
FISHBEIN M and AJZEN I (1975) Belief, Attitude, Intention and Behavior: An
Introduction to Theory and Research. Addison-Wesley, Reading.
FITZGERALD G and RUSSO NL (2005) The turnaround of the London
ambulance service computer-aided dispatch system (LASCAD).
European Journal of Information Systems 14(3), 244–257.
FRASER SG and SALTER G (1995) A motivational view of information
systems success: a reinterpretation of DeLone & McLean’s model. In
Proceedings of the Sixth Australasian Conference on Information Systems
(PERVAN G and NEWBY M, Eds), p 119, Curtin University of Technology,
Perth, Australia.
GABLE G, SEDERA D and CHAN T (2003) Enterprise systems success: a
measurement model. In Proceedings of the Twenty-Fourth International
Conference on Information Systems (MARCH S, MASSEY A and DEGROSS JI,
Eds), p. 576, Association for Information Systems, Seattle, Washington,
USA.
GATIAN AW (1994) Is user satisfaction a valid measure of system
effectiveness? Information & Management 26(3), 119–131.
GEFEN D (2000) It is not enough to be responsive: the role of cooperative
intentions in MRP II adoption. The DATA BASE for Advances in
Information Systems 31(2), 65–79.
GEFEN D and KEIL M (1998) The impact of developer responsiveness on
perceptions of usefulness and ease of use: an extension of the
technology of the technology acceptance model. The DATA BASE for
Advances in Information Systems 29(2), 35–49.
GELDERMAN M (1998) The relation between user satisfaction, usage of
information systems and performance. Information & Management
34(1), 11–18.
GELDERMAN M (2002) Task difficulty, task variability and satisfaction with
management support systems. Information & Management 39(7),
593–604.
GILL TG (1995) Early expert systems: where are they now?. MIS Quarterly
19(1), 51–81.
GILL TG (1996) Expert systems usage: task change and intrinsic
motivation. MIS Quarterly 20(3), 301–329.
Measuring information systems success Stacie Petter et al 259
European Journal of Information Systems
GOODHUE DL and THOMPSON R (1995) Task-technology fit and individual
performance. MIS Quarterly 19(2), 213–236.
GROVER V, PURVIS R and COFFEY J (2003) Information systems effectiveness.
In IsWorld.org, URL: http://business.clemson.edu/ISE/index.html.
GUIMARAES T and IGBARIA M (1997) Client/server system success: exploring
the human side. Decision Sciences 28(4), 851–876.
GUIMARAES T, YOON Y and CLEVENSON A (1996) Factors important to
expert system success: a field test. Information & Management 30(3),
119–130.
HALAWI LA, MCCARTHY RV and ARONSON JE (2007) An empirical
investigation of knowledge-management systems’ success. The Journal
of Computer Information Systems 48(2), 121–135.
HEDGES LV and OLKIN I (1980) Vote-counting methods in research
synthesis. Psychological Bulletin 88(2), 359–369.
HEO J and HAN I (2003) Performance measure of information systems (IS)
in evolving computing environments: an empirical investigation.
Information & Management 40(4), 243–256.
HONG W, THONG JYL, WONG W-M and TAM K-Y (2001/2002) Determinants
of user acceptance of digital libraries: an empirical examination
of individual differences and system characteristics. Journal of Management
Information Systems 18(3), 97–124.
HSIEH JJPA and WANG W (2007) Explaining employees’ extended use of
complex information systems. European Journal of Information Systems
16(3), 216–227.
HWANG MI, WINDSOR JC and PRYOR A (2000) Building a knowledge base
for MIS research: a meta-analysis of a systems success model.
Information Resources Management Journal 13(2), 26–32.
IGBARIA M and TAN M (1997) The consequences of information
technology acceptance on subsequent individual performance. Information
& Management 32(3), 113–121.
IGBARIA M, ZINATELLI N, CRAGG P and CAVAYE ALM (1997) Personal
computing acceptance factors in small firms: a structural equation
model. MIS Quarterly 21(3), 279–305.
IIVARI J (2005) An empirical test of DeLone-McLean model of information
systems success. The DATA BASE for Advances in Information Systems
36(2), 8–27.
IVES B, OLSON M and BAROUDI JJ (1983) The measurement of user
information satisfaction. Communications of the ACM 26(10), 785–793.
JENNEX ME and OLFMAN L (2002) Organizational memory/knowledge
effects on productivity: a longitudinal study. In Proceedings of the
Thirty-Fifth Hawaii International Conference on System Sciences (SPRAGUE JR
RH, Ed), p 109, IEEE Computer Society Press, Big Island, Hawaii, USA.
JIANG JJ, KLEIN G and CARR CL (2002) Measuring information system
service quality: SERVQUAL from the other side. MIS Quarterly 26(2),
145–166.
JONES MC and BEATTY RC (2001) User satisfaction with EDI: an empirical
investigation. Information Resources Management Journal 14(2), 17–26.
KANARACUS C (2008) Gartner: global IT spending growth stable. InfoWorld
April 3, 2008.
KAPLAN RS and NORTON DP (1996) Translating Strategy into Action: The
Balanced Scorecard. Harvard Business School Press, Boston.
KETTINGER WJ and LEE CC (1994) Perceived service quality and user
satisfaction with the information services function. Decision Sciences
25(5), 737–766.
KETTINGER WJ and LEE CC (1997) Pragmatic perspectives on the measurement
of information systems service quality. MIS Quarterly 21(2), 223–240.
KHALIL OEM and ELKORDY MM (1999) The relationship between user
satisfaction and systems usage: empirical evidence from Egypt. Journal
of End User Computing 11(2), 21–28.
KIM J, LEE J, HAN K and LEE M (2002) Business as buildings: metrics for the
architectural quality of internet businesses. Information Systems
Research 13(3), 239–254.
KLEIN R (2007) An empirical examination of patient-physician portal
acceptance. European Journal of Information Systems 16(6), 751–760.
KOSITANURIT B, NGWENYAMA O and OSEI-BRYSON Kweku (2006) An
exploration of factors that impact individual performance in an ERP
environment: an analysis using multiple analytical techniques.
European Journal of Information Systems 15(6), 556–568.
KRAEMER KL, DANZINGER JN, DUNKLE DE and KING JL (1993) The usefulness
of computer-based information to public managers. MIS Quarterly
17(2), 129–148.
KULKARNI UR, RAVINDRAN S and FREEZE R (2006) A knowledge management
success model: theoretical development and empirical validation.
Journal of Management Information Systems 23(3), 309–347.
LASSILA KS and BRANCHEAU JC (1999) Adoption and utilization of
commercial software packages: exploring utilization equilibria, transitions,
triggers and tracks. Journal of Management Information Systems
16(2), 63–80.
LAW CCH and NGAI EWT (2007) ERP systems adoption: an exploratory
study of the organizational factors and impacts of ERP success.
Information & Management 44(4), 418–432.
LECLERCQ A (2007) The perceptual evaluation of information systems
using the construct of user satisfaction: case study of a large French
group. The DATABASE for Advances in Information Systems 38(2), 27–60.
LEONARD-BARTON D and SINHA DK (1993) Developer–user interaction and
user satisfaction in internal technology transfer. Academy of Management
Journal 36(5), 1125–1139.
LI EY (1997) Perceived importance of information system success factors:
a meta-analysis of group differences. Information & Management
32(1), 15–28.
LUCAS HC and SPITLER VK (1999) Technology use and performance: a field
study of broker workstations. Decision Sciences 30(2), 291–311.
MAHMOOD MA, HALL L and SWANBERG DL (2001) Factors affecting
information technology usage: a meta-analysis of the empirical
literature. Journal of Organizational Computing & Electronic Commerce
11(2), 107–130.
MALHOTRA Y and GALLETTA D (2005) Model of volitional systems adoption and
usage behavior. Journal of Management Information Systems 22(1), 117–151.
MARBLE RP (2003) A system implementation study: management commitment
to project management. Information & Management 41(1), 111–123.
MARKUS ML and KEIL M (1994) If we build it, they will come: designing
information systems that people want to use. Sloan Management
Review 35(4), 11–25.
MARTINSONS M, DAVISON MR and TSE D (1999) The balanced scorecard: a
foundation for the strategic management of information systems.
Decision Support Systems 25(1), 71–88.
MCGILL T, HOBBS V and KLOBAS J (2003) User-developed applications and
information systems success: a test of DeLone and McLean’s model.
Information Resources Management Journal 16(1), 24–45.
MCGILL TJ and KLOBAS JE (2005) The role of spreadsheet knowledge in
user-developed application success. Decision Support Systems 39(3),
355–369.
MOLLA A and LICKER PS (2001) E-commerce systems success: an attempt
to extend and respecify the DeLone and McLean model of IS success.
Journal of Electronic Commerce Research 2(4), 131–141.
MORRIS SA, MARSHALL TE and RAINER Jr RK (2002) Impact of user
satisfaction and trust on virtual team members. Information Resources
Management Journal 15(2), 22–30.
MYERS BL, KAPPELMAN LA and PRYBUTOK VRA (1997) Comprehensive model
for assessing the quality and productivity of the information systems
function: toward a contingency theory for information systems
assessment. Information Resources Management Journal 10(1), 6–25.
NOLAN RL (1973) Managing the computer resource: a stage hypothesis.
Communications of the ACM 16(7), 399–405.
OLIVER LW (1987) Research integration for psychologists: an overview of
approaches. Journal of Applied Social Psychology 17(10), 860–874.
PALMER J (2002) Web site usability, design and performance metrics.
Information Systems Research 13(1), 151–167.
PAYTON FC and BRENNAN PF (1999) How a community health information
network is really used. Communications of the ACM 42(12), 85–89.
PITT LF, WATSON RT and KAVAN CB (1995) Service quality: a measure of
information systems effectiveness. MIS Quarterly 19(2), 173–187.
PREMKUMAR G, RAMAMURTHY K and NILAKANTA S (1994) Implementation of
electronic data interchange: an innovation diffusion perspective.
Journal of Management Information Systems 11(2), 157.
RAI A, LANG SS and WELKER RB (2002) Assessing the validity of IS success
models: an empirical test and theoretical analysis. Information Systems
Research 13(1), 5–69.
RAINER RKJ and WATSON HJ (1995) The keys to executive information
system success. Journal of Management Information Systems 12(2), 83.
RIVARD S, POIRIER G, RAYMOND L and BERGERON F (1997) Development of a
measure to assess the quality of user-developed applications. The DATA
BASE for Advances in Information Systems 28(3), 44–58.
RUBIN H (2004) Into the light. In CIO Magazine. http://www.cio.com.au/
index.php/id;1718970659, accessed on July 2004.
SABHERWAL R, JEYARAJ A and CHOWA C (2006) Information systems success:
individual and organizational determinants. Management Science
52(12), 1849–1864.
SCHEEPERS R, SCHEEPERS H and NGWENYAMA OK (2004) Contextual
influences on user satisfaction with mobile computing: findings from
two healthcare organizations. European Journal of Information Systems
15(3), 261–268.
Measuring information systems success Stacie Petter et al260
European Journal of Information Systems
SEDDON PB (1997) A respecification and extension of the DeLone
and McLean model of IS success. Information Systems Research 8(3),
240–253.
SEDDON PB and KIEW M-Y (1996) A partial test and development of
DeLone and McLean’s model of IS success. Australian Journal of
Information Systems 4(1), 90–109.
SEDDON PB, STAPLES S, PATNAYAKUNI R and BOWTELL M (1999) Dimensions
of information systems success. Communications of the Association for
Information Systems 2, 2–39.
SEDDON P, GRAESER V and WILCOCKS LP (2002) Measuring organizational IS
effectiveness: an overview and update of senior management
perspectives. The DATA BASE for Advances in Information Systems
33(2), 11–28.
SEDDON P and YIP S-K (1992) An empirical evaluation of user information
satisfaction (UIS) measures for use with general ledger accounting
software. Journal of Information Systems 6(1), 75–98.
SEDERA D, GABLE G and CHAN T (2004) A factor and structural equation
analysis of the enterprise systems success measurement model. In
Proceedings of the Twenty-Fifth International Conference on Information
Systems (APPELGATE L, GALLIERS R and DEGROSS JI, Eds), p 449, Association
for Information Systems, Washington, DC, USA.
SEGARS AH and GROVER V (1993) Re-examining perceived ease of use
and usefulness: a confirmatory factor analysis. MIS Quarterly 17,
517–522.
SHAW NC, DELONE WH and NIEDERMAN F (2002) Sources of dissatisfaction
in end-user support: an empirical study. The DATA BASE for Advances in
Information Systems 33(2), 41–56.
SHIH HP (2004) Extended technology acceptance model of internet
utilization behavior. Information & Management 41(6), 719–729.
SKOK W, KOPHAMEL A and RICHARDSON I (2001) Diagnosing information
systems success: importance-performance maps in the health club
industry. Information & Management 38(7), 409–419.
STRAUB DW, LIMAYEN M and KARAHANNA-EVARISTO E (1995) Measuring
system usage: implications for IS theory testing. Management Science
41(8), 1328–1342.
SUBRAMANIAN GH (1994) A replication of perceived usefulness and
perceived ease-of-use measurement. Decision Sciences 25(5/6),
863–874.
SUH K, KIM S and LEE J (1994) End-user’s disconfirmed expectations and
the success of information systems. Information Resources Management
Journal 7(4), 30–39.
TENG JTC and CALHOUN KJ (1996) Organizational computing as a
facilitator of operational and managerial decision making. Decision
Sciences 6(2), 673–710.
TEO TSH and WONG I (1998) An empirical study of the performance
impact of computerization in the retail industry. Omega 26(5),
611–621.
THONG JYL, YAP C-S and RAMAN KS (1994) Engagement of external
expertise in information systems implementations. Journal of Management
Information Systems 11(2), 209–231.
THONG JYL, YAP C-S and RAMAN KS (1996) Top management support,
external expertise and information systems implementation in small
businesses. Information Systems Research 7(2), 248–267.
TORKZADEH G and DOLL WJ (1999) The development of a tool for
measuring the perceived impact of information technology on work.
Omega 27(3), 327–339.
VAN DER HEIJDEN H (2004) User acceptance of hedonic information
systems. MIS Quarterly 28(4), 695–704.
VAN DYKE TP, KAPPELMAN LA and PRYBUTOK VR (1997) Measuring
information systems service quality: concerns on the use of the
SERVQUAL questionnaire. MIS Quarterly 21(2), 195–208.
VENKATESH V and DAVIS FD (2000) A theoretical extension of the
technology acceptance model: four longitudinal field studies. Management
Science 46(2), 186–204.
VENKATESH V and MORRIS MG (2000) Why don’t men ever stop to ask for
directions? Gender, social influence, and their role in technology
acceptance and usage behavior. MIS Quarterly 24(1), 115–149.
VENKATESH V, MORRIS MG, DAVIS GB and DAVIS FD (2003) User acceptance
of information technology: toward a unified view. MIS Quarterly 27(3),
425–478.
VLAHOS GE and FERRATT TW (1995) Information technology use by
managers in Greece to support decision making: amount,
perceived value, and satisfaction. Information & Management 29(6),
305–315.
VLAHOS GE, FERRATT TW and KNOEPFLE G (2004) The use of computerbased
information systems by German managers to support decision
making. Information & Management 41(6), 763–779.
WEILL P and VITALE M (1999) Assessing the health of an information
systems portfolio: an example from process engineering. MIS Quarterly
23(4), 601–624.
WINTER SJ, CHUDOBA KM and GUTEK BA (1998) Attitudes toward
computers: when do they predict computer use?. Information &
Management 34(5), 275–284.
WIXOM BH and TODD PA (2005) A theoretical integration of user
satisfaction and technology acceptance. Information Systems Research
16(1), 85–102.
WIXOM BH and WATSON HJ (2001) An empirical investigation of the
factors affecting data warehousing success. MIS Quarterly 25(1),
17–41.
WU J-H and WANG Y-M (2006) Measuring KMS success: a respecification
of the DeLone and McLean model. Information & Management 43(6),
728–739.
YANG HD and YOO Y (2004) It’s all about attitude: revisiting the
technology acceptance model. Decision Support Systems 38(1),
19–31.
YOON Y and GUIMARAES T (1995) Assessing expert systems impact on
users’ jobs. Journal of Management Information Systems 12(1),
225–249.
YOON Y, GUIMARAES T and O’NEAI Q (1995) Exploring the factors
associated with expert system success. MIS Quarterly 19(1), 83–106.
YUTHAS K and YOUNG ST (1998) Material matters: assessing the
effectiveness of materials management IS. Information & Management
33(3), 115–124.
ZHU K and KRAEMER KL (2005) Post-adoption variations in usage and value
of e-business by organizations: cross-country evidence from the retail
industry. Information Systems Research 16(1), 61–84.
Appendix A
Study listing
In addition to the papers cited in the reference list, the
following references were also examined as part of our review
of the literature.
Conceptual papers These manuscripts discuss the D&M
model or IS success in a conceptual manner only with no
data collection or analysis.
1. ARNOLD V (1995) Discussion of an experimental evaluation
of measurements of information system effectiveness.
Journal of Information Systems 9(2), 85–91.
2. AU N, NGAI EWT and CHENG TCE (2002) A critical review of
end-user information system satisfaction research and a
new research framework. Omega 30(6), 451–478.
3. CRONK MC and FITZGERALD EP (1999) Understanding 00
IS
business value00
: derivation of dimensions. Logistics Information
Management 12(1/2), 40–49.
4. CULE PE and SENN JA (1995) The evolution from ICIS 1980
to AIS 1995: have the issues been addressed? In Proceedings
of the Inaugural Americas Conference on Information Systems,
p 15, Association for Information Systems, Pittsburgh,
Pennsylvania, USA.
5. HWANG MI and THORN RG (1999) The effect of user
engagement on systems success: a meta-analytical integration
of research findings. Information & Management 35(4),
229–236.
6. JENNEX ME, OLFMAN L, PITUMA P and YONG-TAE P (1998) An
organizational memory information systems success
Measuring information systems success Stacie Petter et al 261
European Journal of Information Systems
model: an extension of DeLone and McLean0
s I/S success
model. In Proceedings of the Thirty-First Hawaii International
Conference on System Sciences, p 157, IEEE Computer Society
Press, Kohala Coast, Hawaii, USA.
7. KENDALL JE (1997) Examining the relationship between
computer cartoons and factors in information systems use,
success, and failure: visual evidence of met and unmet
expectations. The DATA BASE for Advances in Information
Systems 28(2), 113–126.
8. WOODROOF JB and KASPER GM (1998) A conceptual
development of process and outcome user satisfaction.
Information Resources Management Journal 11(2), 37–43.
9. YOON S (1996) User commitment as an indicator of
information technology use. In Proceedings of the Second
Americas Conference on Information Systems (CAREY JM, Ed),
p 16, Association for Information Systems, Phoenix,
Arizona, USA.
Secondary data literature review These studies were not
considered in the pairwise comparisons because they used
secondary data or conducted a literature review (qualitative
and/or meta-analysis).
1. BROWN RM, GATIAN AW and HICKS Jr JO (1995) Strategic
information systems and financial performance. Journal of
Management Information Systems 11(4), 215–248.
2. BRYNJOLFSSON E (1993) The productivity paradox of information
technology. Communications of the ACM 36(12),
66–77.
3. KETTINGER WJ, GROVER V, GUHA S and SEGARS AH (1994)
Strategic information systems revisited: a study in sustainability
and performance. MIS Quarterly 18(1), 31–58.
Other studies The following studies were not included in
the discussion of pairwise comparisons because the manu-
script:
 only examined a single construct within the D&M model;
 examined antecedents of one or more constructs in the
D&M model;
 phenomenon under study is not an information system
(for example, PC use);
 focused on an alternative model that is not consistent
with variables and constructs within the D&M model;
 performed instrument development only or focused on
measuring constructs;
 non-related variable was hypothesized to moderate relationship
among D&M model variables;
 duplicate of a study already examined and considered
within the literature review.
1. AMOAKO-GYAMPAH K and WHITE KB (1993) User involvement
and user satisfaction – an exploratory contingency
model. Information & Management 25, 1–10.
2. BARKI H and HARTWICK J (1994) User participation,
conflict, and conflict resolution: the mediating roles of
influence. Information Systems Research 6(1), 3–23.
3. BELANGER F, COLLINS RW and CHENEY PH (2001) Technology
requirements and work group communication for telecommuters.
Information Systems Research 12(2), 155–176.
4. BRYNJOLFSSON E and HITT LM (1998) Beyond the productivity
paradox: computers are the catalyst for bigger
changes. Communications of the ACM 41(8), 49–55.
5. BURTON FG, CHEN Y, GROVER V and STEWART KA (1992) An
application of expectancy theory for assessing user
motivation to utilize and expert system. Journal of
Management Information Systems 9(3), 183–198.
6. BYRD TA (1992) Implementation and use of expert systems
in organizations: perceptions of knowledge engineers.
Journal of Management Information Systems 8(4), 97–116.
7. CHAN YE (2000) IT value: the great divide between
qualitative and quantitative and individual and organizational
measures. Journal of Management Information
Systems 16(4), 225–262.
8. CHATTERJEE D, GREWAL R and SAMBAMURTHY V (2002)
Shaping up for e-commerce: institutional enablers of
the organizational assimilation of Web technologies. MIS
Quarterly 26(2), 65–89.
9. COE LR (1996) Five small secrets to systems success.
Information Resources Management Journal 9(4), 29–38.
10. DEVARAJ S and KOHLI R (2000) Information technology
payoff in the healthcare industry: a longitudinal study.
Journal of Management Information Systems 16(4), 41–67.
11. DOS SANTOS BL, PEFFERS K and MAUER DC (1993) The impact
of information technology investment announcements
on the market value of the firm. Information Systems
Research 4(1), 1–23.
12. DUNN CL and GRABSKI S (2001) An investigation of
localization as an element of cognitive fit in accounting
model representations. Decision Sciences 32(1), 55–94.
13. ESSEX PA, MAGAL SR and MASTELLER DE (1998) Determinants
of information center success. Journal of Management
Information Systems 15(2), 95–117.
14. ETEZADI-AMOLI J and FARHOOMAND, AF (1996) A structural
model of end user computing satisfaction and user
performance. Information & Management 30(2), 65–73.
15. FORGIONNE G and KOHLI R (2000) Management support
system effectiveness: further empirical evidence. Journal
of the Association for Information Systems 1 (1es).
16. GALLETTA DF, AHUJA M, HARTMAN A, THOMPSON T and PEACE
AG (1995) Social influence and end-user training.
Communications of the ACM 38(7), 70–79.
17. GOODHUE DL (1995) Understanding user evaluations
of information systems. Management Science 41(12),
1827–1844.
18. HARDGRAVE BC, WILSON RL and EASTMAN K (1999) Toward a
contingency model for selecting an information system
prototyping strategy. Journal of Management Information
Systems 16(2), 113–136.
19. HUNTON JE and BEELER JD (1997) Effects of user participation
in systems development: a longitudinal field
experiment. MIS Quarterly 21(4), 359–388.
20. ISHMAN MD (1996) Measuring information success at the
individual level in cross-cultural environments. Information
Resources Management Journal 9(4), 16–28.
21. JELASSI T and FIGON O (1994) Competing through EDI at
Brun Passot: achievements in France and ambitions for
the single European market. MIS Quarterly 18(4), 337–
352.
22. JONES MC and BEATTY RC (1996) User satisfaction with
EDI: an empirical investigation. In Proceedings of the
Second Americas Conference on Information Systems (CAREY
JM, Ed.), p 191, Association for Information Systems,
Phoenix, Arizona, United States.
23. JURISON J (1996) The temporal nature of IS benefits: a
longitudinal study. Information & Management 30(2), 75–79.
Measuring information systems success Stacie Petter et al262
European Journal of Information Systems
24. KARAHANNA E, STRAUB DW and CHERVANY NL (1999)
Information technology adoption across time: a crosssectional
comparison of pre-adoption and post-adoption
beliefs. MIS Quarterly 23(2), 183–213.
25. LAU F (2001) Experiences from health information system
implementation projects reported in Canada between
1991 and 1997. Journal of End User Computing 13(4), 17–25.
26. LAWRENCE M and LOW G (1993) Exploring individual user
satisfaction within user-led development. MIS Quarterly
17(2), 195–208.
27. LIU C and ARNETT KP (2000) Exploring the factors associated
with Web site success in the context of electronic
commerce. Information & Management 38(1), 23–33.
28. MCHANEY R and CRONAN TP (1998) Computer simulation
success: on the use of the end-user computing satisfaction
instrument: a comment. Decision Sciences 29(2),
525–536.
29. MCKEEN JD and GUIMARAES T (1997) Successful strategies
for user participation in systems development. Journal of
Management Information Systems 14(2), 133–150.
30. MCKEEN JD, GUIMARAES T and WETHERBE JC (1994) The
relationship between user participation and user satisfaction:
an investigation of four contingency factors. MIS
Quarterly 18(4), 427–451.
31. MCKINNEY V, YOON K and ZAHEDI F (2002) The measurement
of Web-customer satisfaction: an expectation and
disconfirmation approach. Information Systems Research
13(3), 296–315.
32. MENON NM, LEE B and ELDENBURG L (2000) Productivity of
information systems in the healthcare industry. Information
Systems Research 11(1), 83–92.
33. MIRANI R and LEDERER AL (1998) An instrument for
assessing the organizational benefits of IS projects.
Decision Sciences 29(4), 803–838.
34. MITRA S. and CHAYA AK (1996) Analyzing cost-effectiveness
of organizations: the impact of information technology
spending. Journal of Management Information
Systems 13(2), 29–57.
35. MUKHOPADHYAY T, RAJIV S and SRINIVASAN K (1997)
Information technology impact on process output and
quality. Management Science 43(12), 1645–1659.
36. NAULT BR and DEXTER AS (1995) Added value and pricing
with information technology. MIS Quarterly 19(4), 449–464.
37. NELSON KM and COOPRIDER JG (1996) The contribution of
shared knowledge to IS group performance. MIS Quarterly
20(4), 409–429.
38. NICOLAOU AI, MASONER MM and WELKER RB (1995)
Intent to enhance information systems as a function of
system success. Journal of Information Systems 9(2), 93–108.
39. PEGELS CC, RAO HR, SALAM AF, HWANG KT and SETHI V (1996)
Impact of IS planning and training on involvement and
implementation success of EDI systems. In Proceedings of
the Second Americas Conference on Information Systems
(CAREY JM, Ed), p 188, Phoenix, Arizona, United States.
40. RAVICHANDRAN T and RAI A (1999) Total quality management
in information systems development: key constructs
and relationships. Journal of Management
Information Systems 16(3), 119–155.
41. SAARINEN T (1996) An expanded instrument for evaluating
information systems success. Information & Management
31(2), 103–119.
42. SABHERWAL R (1999) The relationship between information
system planning sophistication and information
system success: an empirical assessment. Decision Sciences
30(1), 137–167.
43. SAUNDERS CS and JONES JW (1992) Measuring performance
of the information systems function. Journal of Management
Information Systems 8(4), 63–82.
44. SEDDON PB and KIEW M-Y (1994) A partial test and
development of the DeLone and McLean model of IS
success. In Proceedings of the Fifteenth International
Conference on Information Systems (DEGROSS J, HUFF S,
MUNRO MC, Eds), p 99, Association for Information
Systems, Vancouver, Canada.
45. SHARMA S and RAI A (2000) Case deployment in IS
organizations. Communications of the ACM 43(1),
80–88.
46. STAPLES DS, WONG I and SEDDON P (2002) Having
expectations of information systems benefits that match
received benefits: does it really matter? Information &
Management 40(2), 115–131.
47. SUBRAMANI MR and HENDERSON JC (1996) Gaps that matter:
the influence of convergence in perspectives on IS
service quality. In Proceedings of the Second Americas
Conference on Information Systems (CAREY JM, Ed), p 57,
Phoenix, Arizona, USA.
48. SZAJNA B (1993) Determining information system usage:
some issues and examples. Information & Management
25(3), 147–154.
49. TALLON PP, KRAEMER KL and GURBAXANI V (2000) Executives’
perceptions of the business value of information technology:
a process-oriented approach. Journal of Management
Information Systems 16(4), 145–173.
50. TAM KY (1998) The impact of information technology
investments on firm performance and evaluation: evidence
from newly industrialized economies. Information
Systems Research 9(1), 85–98.
51. TAYLOR S and TODD P (1995) Assessing IT usage: the role of
prior experience. MIS Quarterly 19(4), 561–570.
52. TAYLOR S and TODD PA (1995) Understanding information
technology usage: a test of competing models. Information
Systems Research 6(2), 144–176.
53. TENG JTC, JEONG SR and GROVER V (1998) Profiling
successful reengineering projects. Communications of the
ACM 41(6), 96–102.
54. TEO H-H, TAN BCY and WEI K-K (1997) Organizational
transformation using electronic data interchange: the
case of TradeNet in Singapore. Journal of Management
Information Systems 13(4), 139–165.
55. THOMPSON RL, HIGGINS CA and HOWELL JM (1994)
Influence of experience on personal computer utilization:
testing a conceptual model. Journal of Management
Information Systems 11(1), 167–187.
56. VANDENBOSCH B and GINZBERG MJ (1996/1997) Lotus notes
and collaboration: plus ca change. Journal of Management
Information Systems 13(3), 65–81.
57. VENKATESH V and DAVIS FD (1996) A model of the
antecedents of perceived ease of use: development and
test. Decision Sciences 27(3), 451–481.
58. WANG RW and STRONG DM (1996) Beyond accuracy: what
data quality means to data consumers. Journal of Management
Information Systems 12(4), 30–34.
59. YOON Y, GUIMARAES T and CLEVENSON A (1998) Exploring
expert system success factors for business process reengineering.
Journal of Engineering and Technology Management
15(2–3), 179–199.
Measuring information systems success Stacie Petter et al 263
European Journal of Information Systems