Evaluating information systems

Abstract

The book within which this chapter appears is published as a research reference book
(not a coursework textbook) on Management Information Systems (MIS) for seniors
or graduate students in Chinese universities. It is hoped that this chapter, along with
the others, will be helpful to MIS scholars and PhD/Masters research students in
China who seek understanding of several central Information Systems (IS) research
topics and related issues.
The subject of this chapter - ‘Evaluating Information Systems’ - is broad, and cannot
be addressed in its entirety in any depth within a single book chapter. The chapter
proceeds from the truism that organizations have limited resources and those
resources need to be invested in a way that provides greatest benefit to the
organization. IT expenditure represents a substantial portion of any organization’s
investment budget and IT related innovations have broad organizational impacts.
Evaluation of the impact of this major investment is essential to justify this
expenditure both pre- and post-investment. Evaluation is also important to prioritize
possible improvements.
The chapter (and most of the literature reviewed herein) admittedly assumes a blackbox
view of IS/IT1, emphasizing measures of its consequences (e.g. for organizational
performance or the economy) or perceptions of its quality from a user perspective.
This reflects the MIS emphasis – a ‘management’ emphasis rather than a software
engineering emphasis2, where a software engineering emphasis might be on the
technical characteristics and technical performance. Though a black-box approach
limits diagnostic specificity of findings from a technical perspective, it offers many
benefits. In addition to superior management information, these benefits may include
economy of measurement and comparability of findings (e.g. see Part 4 on
Benchmarking IS).
The chapter does not purport to be a comprehensive treatment of the relevant
literature. It does, however, reflect many of the more influential works, and a
representative range of important writings in the area. The author has been somewhat
opportunistic in Part 2, employing a single journal – The Journal of Strategic
Information Systems – to derive a classification of literature in the broader domain. Nonetheless, the arguments for this approach are believed to be sound, and the value
from this exercise real.
The chapter drills down from the general to the specific. It commences with a highlevel
overview of the general topic area. This is achieved in 2 parts: - Part 1
addressing existing research in the more comprehensive IS research outlets (e.g.
MISQ, JAIS, ISR, JMIS, ICIS), and Part 2 addressing existing research in a key
specialist outlet (i.e. Journal of Strategic Information Systems). Subsequently, in Part
3, the chapter narrows to focus on the sub-topic ‘Information Systems Success
Measurement’; then drilling deeper to become even more focused in Part 4 on
‘Benchmarking Information Systems’. In other words, the chapter drills down from
Parts 1&2 Value of IS, to Part 3 Measuring Information Systems Success, to Part 4
Benchmarking IS. While the commencing Parts (1&2) are by definition broadly
relevant to the chapter topic, the subsequent, more focused Parts (3 and 4) admittedly
reflect the author’s more specific interests. Thus, the three chapter foci – value of IS,
measuring IS success, and benchmarking IS - are not mutually exclusive, but, rather,
each subsequent focus is in most respects a sub-set of the former.
Parts 1&2, ‘the Value of IS’, take a broad view, with much emphasis on ‘the business
Value of IS’, or the relationship between information technology and organizational
performance.
Part 3, ‘Information System Success Measurement’, focuses more specifically on
measures and constructs employed in empirical research into the drivers of IS success
(ISS). (DeLone and McLean 1992) inventoried and rationalized disparate prior
measures of ISS into 6 constructs – System Quality, Information Quality, Individual
Impact, Organizational Impact, Satisfaction and Use (later suggesting a 7th construct –
Service Quality (DeLone and McLean 2003)). These 6 constructs have been used
extensively, individually or in some combination, as the dependent variable in
research seeking to better understand the important antecedents or drivers of IS
Success. Part 3 reviews this body of work.
Part 4, ‘Benchmarking Information Systems’, drills deeper again, focusing more
specifically on a measure of the IS that can be used as a ‘benchmark’3. This section
consolidates and extends the work of the author and his colleagues4 to derive a robust,
validated IS-Impact measurement model for benchmarking contemporary Information
Systems (IS). Though IS-Impact, like ISS, has potential value in empirical, causal
research, its design and validation has emphasized its role and value as a comparator;
a measure that is simple, robust and generalizable and which yields results that are as
far as possible comparable across time, across stakeholders, and across differing
systems and systems contexts.