Chapter 8: Diagnostics, Fault & Change Management Flashcards

Chapter 8 Overview and Learning Objectives

• Goal of the Chapter: Understanding how systems fail, recover, and maintain operational quality through diagnostics, fault management, and change management.

Fault Tolerance & Network Topologies

• Fault Tolerance Definition: Ensures continuous system operations despite failures in hardware, software, or network components.

• Redundancy: Resilience is increased through backup links, failover servers, and duplicate systems.

• Network Topologies and Resilience:

  • Centralized: Features a single hub node. It is easy to manage but carries high single-point risk and low resilience.

  • Decentralized / Hierarchical: Features multiple clusters. It offers moderate resilience and is common in enterprise environments.

  • Distributed Mesh: Features many redundant paths. It provides the highest fault tolerance but is complex to administer.

• Principle 43 – Predictable Failure: Systems should fail predictably so they can be recovered quickly. This requires standardized protocols and QA procedures.

• Error Propagation: Networks are "webs of influence" where the risk of error spreading depends on the topology.

  • Propagation occurs when failures spread across dependent systems or connected nodes.

  • Topology impacts fault spread speed, recovery time, and security risks (security breaches propagate faster in poorly segmented or centralized environments).

• Key Insight: Increased fault tolerance usually results in higher implementation costs and management complexity.

Networks and the Small-World Phenomenon

• The Six Degrees Phenomenon: Any two people on Earth are connected by approximately $6$ hops.

• Small-World Networks Characteristics:

  • They are locally clustered yet maintain globally short distances.

  • Weak Links: A few long-range "weak links" provide vital shortcuts. These improve global connectivity but can introduce security vulnerabilities.

  • In regular networks, nodes are mostly connected to nearby nodes.

• Scale-Free Behavior:

  • Preferential Attachment: New nodes tend to connect to already popular nodes (the "rich get richer").

  • This creates hubs that accumulate many connections while most nodes have few.

  • Hub Impact: Hubs improve communication efficiency but become critical points of failure and specific targets for attack.

• Implications for Failures:

  • Fault Propagation Risk: Small-world properties allow fast communication but accelerate fault spread.

  • Security Breach Pathways: Attacks often exploit weak links or central hubs.

  • Emergent Bugs: Emergent properties can create unexpected system behaviors or hidden vulnerabilities.

• Management Strategy: Network segmentation is used to reduce large-scale fault and attack propagation.

Causality, Dependency, and System Definition

• Principle 44 – Causality: Every change or effect happens in response to a cause that precedes it. Root cause analysis seeks the original source of faults rather than symptoms.

• Dependency Dynamics:

  • Subsystems share variables, meaning causal links cross boundaries.

  • Factors only separate if variables fully separate ($S(x_1...x_n)$).

  • Third-party interactions can cause behavior that seems "impossible" at first glance.

  • Data Resolution: Coarse-graining data destroys vital causal information.

• Defining System Boundaries:

  • A boundary is not just a hardware box; it is a probability sphere of all potential fault causes.

  • Boundaries must include users, the network, temperature, policies, and environmental dependencies.

  • Analysis Method: Start analysis beyond the limit of interest, then prune systematically.

• Key Insight on Failure: Computers most often fail due to usage patterns and management rather than hardware failure alone.

Fault Classification and Diagnostics

• IEEE Software Anomalies (Technical Faults):

  • OS/Program crashes and program hang-ups.

  • Input/Output problems.

  • Failed required performance or degraded service (wrong/no output).

  • System error messages.

• Human-Edge Errors:

  • Management error and miscommunication.

  • Psychological factors: Forgetfulness, confusion, stress, ignorance, or carelessness.

  • Inability to cope with complexity.

  • Systematic and random procedural errors.

• Core Administrative Problems: Many issues stem from input errors (misconfiguration) and resource exhaustion, which cannot be fixed by evaluating components in isolation. Administration is both social and technical.

• Principle 45 – Diagnostics: Always eliminate the obvious first. Think "horses, not zebras." Use simple checks for loose cables or incorrect permissions before investigating complex causes.

• Three-Step Diagnosis Process:

  1. Gather Evidence: Collect data from users, log files, system calls, and tests.

  2. Form a Hypothesis: Develop a best-guess probable cause based on evidence.

  3. Test & Reproduce: Confirm the cause by toggling the error "on" and "off" in a controlled manner.

• Administrator Toolbox: Basic OS primitives include the ability to create, edit, replace, rename, or remove files; change access rights; and start, stop, or signal processes.

Cause Trees and Root Cause Analysis (RCA)

• Cause Tree Definition: A visual tool used to track a problem back to its origin to prevent recurrence.

• Structure: Start with the observed fault at the top, branch into immediate causes, and then sub-causes.

• Event Tree Analysis (ETA): A related method mapping all possible "True/False" (binary) pathways an event could take.

• Example: Disk Full Case Tree:

  • Level 1: Disk Full.

  • Level 2 (Immediate Causes): Log files, Tmp files, Legitimate usage, Bad users.

  • Level 3 (Human Management Causes): Inadequate procedures, poor training, lack of quality control, miscommunication, lack of preventive maintenance.

Probabilistic Fault Trees

• Component Fault Categories:

  • Primary Faults: A component fails while operating within its design limits (e.g., a server rated for $50\text{ tx/s}$ fails at $30\text{ tx/s}$).

  • Secondary Faults: A component fails because it is pushed outside its design specifications (e.g., same server fails at $90\text{ tx/s}$).

  • Command Faults: The component functions correctly but is triggered at the wrong time/place (e.g., database queries triggered without user requests).

• Gate Logic and Probability Formulas:

  • OR Gate: Represents a situation where any input triggers the fault. Probability grows, making the system more dangerous.

    • $P(A \text{ or } B) = P(A) + P(B) - P(A \land B)$

  • AND Gate: Represents a situation where all inputs must occur to trigger the fault. Probability shrinks, making the system more secure.

    • $P(A \text{ and } B) = P(A) \times P(B|A)$

  • XOR Gate: Represents exclusive scenarios with no predictable direction for security.

    • $P(A \text{ xor } B) = P(A) + P(B) - 2P(A \land B)$

• Cutsets: The minimal set of basic events that triggers the top-level fault. Identifying these is the primary goal of fault tree analysis.

Change Management Procedures

• Philosophy: Treat changes like "controlled faults" to maintain widespread predictability with minimal disruption.

• Nine Steps of Change Management:

  1. Decide on the change: Clearly define objectives.

  2. Map repercussion network: Identify dependencies and how the change ripples.

  3. Revise policy: Update rules for each affected component.

  4. Inform users: Await comments/feedback.

  5. Incorporate feedback: Adjust policy based on user input.

  6. LOCK system: Prevent partial reconfiguration hazards.

  7. Make changes: Execute the physical or configuration modification.

  8. UNLOCK system: Resume normal operations.

  9. Inform users change is live: Final notification.

Monitoring and Performance Tuning

• Monitoring Levels:

  • Machine performance (hardware/OS).

  • Abstract policy (usage and traffic patterns).

• Tools:

  • SNMP Tools: MRTG, RRDtool, Cricket.

  • Anomaly Detection: Tracking abnormal traffic, kernel patterns, and usage profiles via the Cfengine environment daemon.

• Performance Tuning Layers (Priority Order):

  1. Hardware Optimization: Focus on RAM first.

  2. Kernel Configuration: Tune the operating system core.

  3. Software Tuning: Optimize application-level servers (e.g., WWW, SMTP).

  4. Contention Reduction: Minimize disk and network traffic competition.

• Principle 48: Always fix problems at the root, not by patching symptoms.

• Principle 49 – Weakest Link: System performance is limited by its weakest component. Always optimize at the source first.

Principles of Quality Assurance (QA)

• QA Framework: Based on ISO 9000 and Quality Management systems.

• Key QA Actions:

  • Define Quality Goals: Decide what to accomplish (Policy First).

  • Assess Current State: Establish a baseline measurement before changes.

  • Devise Strategy: Be willing to back-track to a better starting point if necessary.

  • Project Management: Focus on execution and resource budgeting.

  • Documentation & Verification: Essential for error prevention and guideline conformance.

  • Fault-Handling Procedure: QA must be self-assessing and handle exceptions through defined procedures.

• Principle 51 – Rapid Maintenance: Response speed is crucial because the environment changes constantly. Quality is a journey, not a destination.

Chapter 8 Summary of Principles

• P43 Predictable Failure: Systems should fail predictably for fast recovery.

• P44 Causality: Every effect has a preceding cause.

• P45 Diagnostics: Always eliminate the obvious first.

• P46 Strategic Admin: Administration is a strategic game to maintain policy.

• P47 Nash Dilemma: Private interest does not always equal community interest.

• P48 Symptoms & Cause: Fix root causes, not symptoms.

• P49 Weakest Link: Performance is limited by the weakest component.

• P50 Contention: Resource competition can dramatically reduce performance.

• P51 Rapid Maintenance: Speed of response is crucial; quality is an ongoing process.

• Core Takeaway: Plan for failure, diagnose systematically, manage change deliberately, and monitor continuously.