Palo Alto SD-WAN Training

Introduction to Training

  • Disclaimer: Introduced to set context.

  • Purpose of Training: Enhance troubleshooting skills and knowledge of SD-WAN technology.

Overview of SD-WAN Technologies

  • Major technologies discussed:

    • VeloCloud: Primary technology used.

    • Other Technologies: Versa, Palo Alto, Meraki, Fortinet.

  • Palo Alto:

    • Known for top-tier cybersecurity firewall setups.

    • Noted challenges compared to other technologies like Fortinet.

Device and Hardware Overview

  • Series Overview:

    • 1,200 Series: Typically for branch locations.

    • 3,000 Series: Heavy-duty for larger setups.

    • 9,000 Series: Designed for data centers and hub locations.

  • Hardware Controllers:

    • Various controllers based on the deployment:

    • Versa Directors

    • VeloCloud Orchestrator for VMware machines

    • Palo Alto: Strata Cloud Manager (SCM)

    • Edge control devices - IONs.

Troubleshooting Overview

  • General Approach:

    • Similar process for P2 and P3 tickets.

    • Focus on alert checks and prechecks before triage and troubleshooting.

  • Triage and Troubleshooting Dependencies:

    • Triage based on Region of Engagement (ROE).

    • Importance of contacting clients through proper channels (phone/email).

    • Use of a cheat sheet for engaging AMS clients.

Specific Issues Covered

  • Common Issues Discussed:

    • BGP (Border Gateway Protocol) down

    • Interfaces being down

    • Disconnection from controllers.

BGP (Border Gateway Protocol) Troubleshooting

  • Alerts:

    • Recurring alerts for BGP being down; potential reasons include:

    • Device state changes (active to backup)

    • Link flap or hard down situations

    • Carrier degradation affecting tunnel builds.

  • Workflow for Troubleshooting BGP:

    1. Identify and acknowledge alerts regarding BGP status.

    2. Perform prechecks including CI identity and serial number.

    3. Use reference materials from the Knowledge Base (KB).

    4. Capture and document troubleshooting steps and alerts.

Accessing Support Features

  • Login Procedures:

    • Access to SharePoint for resources and help from engineering teams.

  • Capture Tools:

    • Using tools for capturing alerts and screenshots for documentation.

    • Example noted: Capture options in Google Chrome.

Incident Documentation and Handling

  • Incident Templates:

    • Importance of structuring incident reports accurately.

  • Incident Classification:

    • Use of timestamps, severity ratings for issues (high impact).

    • Triage includes assessing system ARP and routing statuses.

Training Effectiveness and Questions

  • Open discussions regarding the training effectiveness and participant feedback.

  • Share observations and coping strategies regarding the systems and troubleshooting experiences in SD-WAN.

Technical Problem-Solving Practices

  • Discussed engagement levels with different tiers (Tier 1, Tier 2, Tier 3) of technical support.

  • Importance of reaching out for help when standard troubleshooting fails.

Final Thoughts and Wrap-Up

  • Even experienced technicians find Palo Alto's SD-WAN solutions intimidating, but consistent practice and familiarity improve comfort levels.

  • Encouragement for ongoing training and familiarity with equipment and systems to enhance response times and effectiveness in problem resolution.

  • Logistics for next training sessions confirmed, along with expectations for future interactions.

Introduction to Training

  • Disclaimer: This documentation provides a comprehensive guide for internal technical teams to standardize support processes.

  • Purpose of Training: Focused on enhancing high-level troubleshooting skills and advancing technical depth in SD-WAN (Software-Defined Wide Area Network) architectures, specifically targeting performance optimization and uptime.

Overview of SD-WAN Technologies

  • Core Technologies:

    • VeloCloud (VMware): The primary platform utilized; known for Gateways, Orchestrators, and Edges using Dynamic Multi-Path Optimization (DMPO).

    • Versa Networks: Utilized for software-defined security and routing integrated into a single stack.

    • Palo Alto (Prisma SD-WAN): High-performance solution prioritizing application-defined networking.

    • Meraki (Cisco): Cloud-managed IT solution focusing on ease of deployment.

    • Fortinet: Known for Secure SD-WAN that integrates NGFW (Next-Generation Firewall) capabilities.

  • Palo Alto Insights:

    • Renowned for industry-leading cybersecurity firewall integration.

    • Complexity: Noted for having a steeper learning curve and more complex configuration workflows compared to competitors like Fortinet, particularly regarding security policy inheritance.

Device and Hardware Overview

  • Hardware Series and Use Cases:

    • 1,200 Series: Compact form factors designed for small branch offices or SOHO (Small Office/Home Office) locations with limited throughput requirements.

    • 3,000 Series: Mid-range, heavy-duty hardware equipped with higher processing power for medium-to-large branch environments.

    • 9,000 Series: High-capacity appliances designed specifically for Data Centers, Regional Hubs, and high-throughput aggregation points.

  • Control Plane and Management:

    • Versa Directors: The centralized management platform for Versa nodes.

    • VeloCloud Orchestrator (VCO): A multitenant, cloud-based management system for VMware SD-WAN.

    • Palo Alto Strata Cloud Manager (SCM): The AI-powered management plane for controlling Prisma SD-WAN and SASE components.

    • ION (Instant-On Network): The edge hardware devices specifically used in Palo Alto SD-WAN deployments.

Troubleshooting Methodology

  • General Triage Approach:

    • P2P2 and P3P3 Tickets: Incident management follows a severity-based protocol. P2P2 represents significant service degradation, while P3P3 focuses on minor issues or single-user impacts.

    • Pre-Triage Check: Verification of power status, physical cabling, and environmental alerts before moving to logic/routing layers.

  • Dependency and Engagement:

    • Region of Engagement (ROE): Troubleshooting steps must align with the specific geographic and contractual ROE requirements.

    • Client Communication: Mandatory outreach via phone and email. Support staff must utilize the official AMS (Asset Management System) cheat sheet for client-specific contact protocols and escalation paths.

BGP (Border Gateway Protocol) Deep Dive

  • Alert Indicators: Frequent alerts occur when BGP adjacencies fail. Key causes include:

    • HA State Changes: Failover events where the primary device shifts to backup, causing a sub-second or multi-second neighbor reset.

    • Link Flapping: Rapid intermittent connectivity on the physical or sub-interface causing the BGP finite state machine to cycle through "Idle" and "Active."

    • Carrier Degradation: Latency or packet loss on the underlying transport (ISP) that prevents BGP Keepalive packets from being received.

  • BGP Troubleshooting Workflow:

    1. Identification: Acknowledge the alert in the monitoring system to prevent redundant ticketing.

    2. Asset Validation: Verify CI (Configuration Item) names and serial numbers in the CMDB (Configuration Management Database).

    3. Knowledge Base (KB) Research: Consult KB articles specifically for "BGP Neighbor Loss" or "Tunnel Down" scenarios.

    4. Diagnostic Capture: Execute show commands (e.g., show ip bgp summary) and capture output for the engineering review.

Documentation and Reporting

  • Incident Standards:

    • Reports must include precise timestamps (UTCUTC) and clear severity ratings.

    • System Verification: During triage, technicians must assess ARP (Address Resolution Protocol) tables and routing tables (RIB) to ensure the device is correctly mapping MAC addresses and learning paths.

  • Tools:

    • SharePoint: Central repository for all design documents and engineering handbooks.

    • Capture Utilities: Use of standardized screenshot tools or browser-based capture for visual proof of errors within orchestrator dashboards.

Professional Development and Escalation

  • Technical Tiers: Engagement moves linearly from Tier 1 (Initial Intake) to Tier 2 (Specialized Support) and Tier 3 (Advanced Engineering/Vendor Escalation).

  • Conclusion: Continuous exposure to complex systems like Palo Alto decreases the "intimidation factor." Familiarity with the dashboard and hardware layouts is the primary driver for reducing Mean Time to Repair (MTTRMTTR).

  1. Initial Foundation and Preliminary Research

    • Acknowledge Scope: Internal teams must follow this standardized guide to ensure consistent support across all SD-WAN architectures.

    • Identify Performance Goals: The objective of the troubleshooting process is to maximize performance optimization and uptime.

  2. Platform and Vendor Identification

    • Determine Core Technology: Identify which SD-WAN vendor is utilized at the site:

      • VeloCloud (VMware): Focuses on Gateways and Orchestrators via Dynamic Multi-Path Optimization (DMPO).

      • Palo Alto (Prisma SD-WAN): High-performance, application-defined networking requires a specialized approach due to security policy inheritance complexity.

      • Versa Networks: Look for integrated software-defined security and routing in a single stack.

      • Cisco Meraki: Cloud-managed solutions usually chosen for ease of deployment.

      • Fortinet: Secure SD-WAN that integrates Next-Generation Firewall (NGFW) functions.

  3. Hardware Assessment and Deployment Classification

    • Evaluate Hardware Series: Match the encountered device to its intended environment:

      • 1,200 Series: Small branch or SOHO (Small Office/Home Office).

      • 3,000 Series: Mid-to-large branch environments requiring high processing power.

      • 9,000 Series: Data Centers, high-capacity Regional Hubs, and aggregation points.

    • Verify Control Plane Tools: Access the specific management interface:

      • VeloCloud Orchestrator (VCO) for VMware.

      • Strata Cloud Manager (SCM) for Palo Alto ION (Instant-On Network) devices.

      • Versa Directors for Versa nodes.

  4. Incident Triage and Severity Classification

    • Assign Severity Level:

      • P2P2 Tickets: Represent significant service degradation affecting critical business flows.

      • P3P3 Tickets: Focus on minor issues or single-user impacts.

    • Perform High-Level Pre-Triage:

      • Check physical cabling and power status.

      • Review environmental alerts (temperature, fan status).

      • Confirm the Region of Engagement (ROE) to align with contractual troubleshooting requirements.

  5. BGP (Border Gateway Protocol) Failure Resolution

    • Identify Root Cause Indicators: Analyze why BGP adjacencies have failed:

      • High Availability (HA) State Changes: Check for device failovers from primary to backup.

      • Link Flapping: Look for rapid intermittent connectivity on physical or sub-interfaces causing the BGP state machine to cycle.

      • Carrier Degradation: Analyze underlying ISP transport for latency or packet loss preventing Keepalive receipt.

    • Execute Troubleshooting Workflow:

    1. Acknowledge monitoring alerts to avoid duplicate ticket generation.

    2. Validate Configuration Item (CI) names and serial numbers in the CMDB.

    3. Search the Knowledge Base (KB) for "Neighbor Loss" or "Tunnel Down" scenarios.

  6. Technical Data Capture and RIB Analysis

    • Run Diagnostic Commands: Execute CLI or Orchestrator commands like show ip bgp summary.

    • Assess Routing Tables (RIB): Verify that the device is learning appropriate paths and correctly mapping MAC addresses in the Address Resolution Protocol (ARP) tables.

    • Visual Proof: Use browser-based capture tools for snapshots of Orchestrator dashboards during error states.

  7. Documentation, Reporting, and Final Escalation

    • Standardize Incident Reports:

      • Use precise UTCUTC timestamps.

      • Reference design documents stored in SharePoint.

    • Follow Tiered Escalation: If standard troubleshooting fails, escalate linearly:

      • Tier 1: Initial Intake and validation.

      • Tier 2: Specialized support and investigation.

      • Tier 3: Advanced Engineering and Vendor Escalation.

    • Post-Incident Review: Familiarity with dashboard layouts and hardware reduces the overall Mean Time to Repair (MTTRMTTR).