Chapter 5a: Evaluation of Health Information Systems: Purposes, Theories, and Methods

Core Definition and Purpose of HIS Evaluation

• Health Information System ($HIS$) evaluation is defined as the act of measuring or exploring system properties during planning, development, implementation, or operation to inform decisions in specific contexts (Ammenwerth et al., 2004).
• The full potential of an $HIS$ is reached only when development is accompanied by rigorous evaluation throughout its entire life cycle.
• Evaluators must distinguish whether their purpose is to determine how a product should be built, how it should be implemented, or the value of the completed product.

Socio-technical Levels in Health Informatics

• Social Level: Encompasses the inner context, such as organizational culture and workflows, and the outer context, such as patients and government regulations.
• Human Level: Focuses on human factors, including the characteristics, needs, and limitations of users. Usage failure often occurs when human factors are ignored.
• Technical Level: Relates to the software itself, including its functionality, architecture, and technology standards.

The HIS Life Cycle and Evaluation Types

• Planning Phase: Focuses on idea conception, prioritization, requirements gathering, and project governance review.
• Development Phase: Involves software design, prototyping, and organizational approval.
• Implementation Phase: Includes implementation strategy design, staff training, and pilot rollouts followed by iterative improvements.
• Operation Phase: Involves maintenance and monitoring after software changes have stabilized and the system may be disseminated to other sites.
• Formative Evaluation: Provides feedback for continuous program improvement.
• Summative Evaluation: Assesses the merit, properties, and outcomes of a program, typically during the operation phase.

Hierarchy of Theoretical Perspectives

• Theories: The most complex level with the widest scope; they aim to explain mechanisms of action (e.g., specifying relations among variables to predict phenomena).
• Frameworks: Aim to describe factors influencing outcomes and specify how to measure those determinants.
• Models: The most basic and practical level; they are often program-specific representations of goals and processes.

Specialized Frameworks and Models by Phase

• Planning and Planning Models:
  • Program Logic Models: Representations of program components including Inputs, Activities, Outputs, and Outcomes.
  • Situation Awareness ($SA$) Framework: Developed by Endsley, it defines three levels of performance: Perception ($Level \, 1$), Comprehension ($Level \, 2$), and Projection ($Level \, 3$).
  • Naturalistic Decision Making ($NDM$): Explores cognitive functions and human behavior in real-world contexts ($Klein, 1997$).
• Development and Adoption Models:
  • Plan-Do-Check-Act ($PDCA$): An iterative model for quality improvement and software prototyping.
  • Unified Theory of Acceptance and Use of Technology ($UTAUT$): Predicts intention to use a system based on performance expectancy, effort expectancy, social influence, and facilitating conditions. It explains approximately $70\%$ of the variance in usage intention.
• Implementation Frameworks:
  • Diffusion of Innovations ($Rogers, 1983$): Adoption rates are driven by innovation attributes: relative advantage, compatibility, complexity, trialability, and observability.
  • Normalization Process Theory ($NPT$): Focuses on coherence, cognitive participation, collective action, and reflexive monitoring to embed innovations into social contexts.
  • RE-AIM: An evaluation framework measuring Reach, Effectiveness, Adoption, Implementation, and Maintenance.
• Operational and Success Models:
  • Structure-Process-Outcome Model ($Donabedian, 1988$): Categorizes healthcare quality measures into structural resources, care-delivery tasks, and final measurable outcomes.
  • DeLone and McLean Information Systems Success Theory: Evaluates system quality, information quality, service quality, and net benefits.