Section 25: Troubleshooting Network Services

MAC Address

• 12-digit hexadecimal number used to uniquely identify a NIC on a network

  • 48 bits in total length

    • First 24 bits: assigned by the hardware manufacturer

    • Next 24 bits: NIC identifier

Duplicate MAC Addresses at Layer 2

• Can cause network issues

  • Causes confusion in switch forwarding tables (CAM tables)

  • Connectivity problems

• MAC Spoofing

  • Using a self-assigned address (locally administered address), can lead to duplicate addresses

• Virtual Machines(VMs)

  • Can also create duplicate MAC addresses

  • Logical Domain Manager

    • Used as preventive solution by monitoring and reassigning MAC addresses

Identifying Duplicate MAC Address

• Network connectivity issues

• Intermittent connectivity for affected devices

• Use of a protocol analyzer like Wireshark to analyze ARP traffic for duplicate MAC address mappings

Preventing and Resolving Duplicate MAC Addresses

• Enable port security on devices to allow only one MAC address per switch port

• Use the ‘show arp’ command on switches to identify switch ports with duplicate MAC addresses

• Check and correct hardware manufacturing issues or MAC spoofing

• Replace NIC if it is a hardware issue

Duplicate IP Addresses at Layer 3

• Duplicate IP Addresses

  • Known as IP address conflict

  • When two devices on the same network have the same IP

  • Causes

    • Static IP assignments

    • DHCP server issues

    • Rogue DHCP servers

Identifying Duplicate IP Addresses

• Check network adapter properties to see if the IP address is statically assigned or obtained dynamically

• Use the ‘show arp’ command on routers to identify duplicate IP addresses

Preventing and Resolving Duplicate IP Addresses

• Correct static IP assignments or switch to dynamic IP assignment if necessary

• Use DHCP server properly and check for rouge DHCP servers

• Verify configurations on individual clients to ensure proper IP assignment

Dynamic Host Configuration Protocol (DHCP)

• A network management protocol on IP networks to automatically assign IP addresses and other communication parameters to devices using a client-server architecture

Rogue DHCP Server

• A DHCP server on the network that is not under administrative control

• Risks

  • Can be installed maliciously to redirect traffic or accidentally by employees

    • Causes IP conflicts and network connectivity issues

• Prevention

  • Configure DHCP snooping to exclude rogue DHCP server traffic

  • Use port security on switch ports

  • Configure an intrusion detection system (IDS)

DHCP Scope Exhaustion

• Occurs when the DHCP server runs out of valid IPs to assign

• Causes

  • Too many devices requesting IPs simultaneously

  • Long lease times

• Solutions

  • Increase the DHCP scope size

  • Decrease lease times for transient users

  • Enable port security or Network Access Control (NAC) to limit the number of devices using DHCP

Routing Issues (3)

• Multicast Flooding

• Asymmetrical Routing

• Missing Routes

Multicast Flooding

• Multicast Networks

  • Send group communications to multiple destination computers simultaneously

• Flooding occurs when no specific host is associated with the multicast MAC address in the switch’s CAM table

• Results in multicast traffic being flooded throughout the LAN or VLAN, wasting resources

• Prevention

  • Configure switches to block unknown multicast packets

Asymmetrical Routing

• Occurs when packets leave via one path and return via a different path

• Can occur across different layer two bridge pair interfaces, routers, or firewalls in a high availability cluster

• Problematic for security devices and network appliances performing deep packet inspection or using stateful firewalls

  • Doesn’t cause any routing issues necessarily, but do cause issues with dropped packet flows

• Solution

  • Adjust firewall placement and internal routing to ensure traffic flows through the same firewall in both directions

Missing Routes

• Occurs when a router cannot reach a destination due to a missing route in the routing table

• Common with static routes if mistyped or not properly added

• Troubleshoot by checking the routing tables

  • show ip route - Cisco

  • route print - Windows

  • For dynamic routing protocols like OSPF or BGP

    • Verify that the dynamic routing protocol is enabled

    • Ensure routers can communicate

Switching Loops

• Occur when there is more than one path between a source and destination device

• Can lead to broadcast storms due to repeated broadcast messages in a looped architecture

• Prevention

  • Enable Spanning Tree Protocol (STP) on switches

    • show spanning tree - check STP config

Routing Loops

• Formed when there is an error in the routing algorithm, creating a circular route

• Caused by incorrect configurations of routing protocols

• Prevention

  • Routing Protocols

    • Have methods in place to prevent physical loops that cause issues

  • Split Horizon

    • Prevents a route from being advertised back in the direction it came from

    • ip split horizon - set up split horizon on Cisco router

  • Route Poisoning

    • Increases the metric of a failed route to an infinitely high number

  • Hold-down timers

    • Prevent bad routes from being restored and passed to other routers

    • Hold-down period default - 180 seconds (3 minutes)

Tips for Loops

• Use correct routing protocols and ensure proper configs to avoid loops

• Be cautious when adding static routes, as they can lead to routing loops if not configured properly

• Static routes are highly trusted by routers

  • Default metric - 1

Firewalls

• Network security devices that monitor and filter incoming and outgoing network traffic based on established rule sets

• Act as an inspection point and barrier between a private internal network and the public internet or other private internal networks

Types of Firewalls (2)

• Host-based

• Network-based

Host-based Firewall

• Software that runs on an individual computer or device, protecting that single device (e.g., Windows Defender firewall)

Network-based Firewall

• A network security device deployed in line with network traffic flow, monitoring and filtering traffic (e.g., Cisco firewall)

Common Firewall Issues

1. Access to protected resources from unprotected networks is not working

2. Access to unprotected resources from protected networks is not working

3. Access to the firewall and its configurations is not working

Firewall Troubleshooting Steps

• 7-Step Troubleshooting Method

• Understand the OSI model to troubleshoot each layer from Layer 1 physical to identify the issue

  • Verify physical connectivity (Layer 1) by checking cables and link lights

  • Check Layer 2 by ensuring communication using ARP and MAC addresses

  • Check Layer 3 for valid IP address, subnet mask, and default gateway

• Inspect firewall for misconfigured rule sets, such as ACLs

Access Control Lists (ACLs)

• Collection of permit and deny conditions providing security by blocking unauthorized users and allowing authorized users

  • show access-lists - Cisco command

• Verify ACL rules for typos, correct protocol and port numbers, source and destination addresses, and rule order

  • Example ACL Troubleshooting

    • Identify ACL rules causing connectivity issues (e.g., denying TCP traffic from any IP to any IP)

    • Adjust ACL rule order to prioritize more specific rules (e.g., moving specific allow rules to the top of the list)

Software Firewall Considerations

• Verify IP addresses, ports, applications, and services are correctly allowed or blocked

• Double-check ACLs to ensure they’re blocking, and allowing exactly what is intended and in the right order

IP Settings

• Incorrect IP settings can cause issues

• Every network client needs four key pieces of information

  • IP Address

  • Subnet Mask

  • Default Gateway IP

  • DNS Server IP

Incorrect IP Troubleshooting Steps

• 1 - Identify the Issue

  • Use ping to test connectivity (e.g., ping 8.8.8.8)

• 2 - Analyze IP Settings

  • Check IP address, subnet mask, and default gateway

    • Ensure that they are correct and in the same subnet

• 3 - Resolve Issues

  • Wrong default gateway

    • Change it to the correct IP address in the same subnet

DNS Configuration

• Ensure DNS server IP addresses are correct

• If no DNS servers are available

  • Use public DNS servers (e.g., google DNS of 8.8.8.8 and 8.8.4.4)

VLAN Communication

• Devices in different VLANs cannot communicate directly

• Routing between VLANS is necessary for communication to occur

• Devices within the same VLAN must belong to the same logical subset

Improper VLAN Configuration

• Can cause devices to be unable to communicate

• Verify VLAN configuration and routing setup to resolve issues

Avoiding Default VLAN

• Not using VLANs places all traffic in the default VLAN (VLAN 1)

  • Leads to a large broadcast domain

• Segregate servers into their own VLANs to improve performance and reduce broadcast issues

DNS Issues

• Matches domain names with corresponding IP addresses

• DNS Issue Symptom

  • Network clients unable to resolve domain names to IPs

• Determine if the issue is on a single client or network-wide

  • Single Client Issue

    • Possible Cause

      • TCP/IP settings on the client

    • Resolution Steps

      • Check assigned DNS server IP

      • Verify connectivity to DNS server

  • Network-Wide DNS Issue

    • Possible Cause

      • DNS server not responding

    • Resolution Steps

      • Flush DNS cache

      • Change to a different DNS server (e.g., Google’s DNS servers at 8.8.8.8 and 8.8.4.4)

DNS Server Troubleshooting

• Issue

  • DNS server not properly responding

• Resolution Steps

  • Verify A records and CNAME records

  • Ensure TTL is set correctly

DNS Records Verification

• A Records

  • Verify domain name and IP address are correct

• CNAME Records

  • Verify source/destination domain names are spelled correctly

• nslookup - command for verification

DNS Time to Live (TTL)

• Issue

  • TTL set too high causing old DNS records to remain cached

• Recommended TTL

  • Keep TTL short (e.g., 300 seconds) for frequent network or website changes

Reducing DNS Latency

• Issue

  • High latency due to distant DNS servers

• Resolution

  • Use DNS servers closer to users, such as those hosted within your network or by your ISP

Troubleshooting NTP Issues

• NTP Purpose

  • Synchronizes system clocks for distributed applications

• Issue

  • NTP packets not received, processed, or contain errors

Troubleshooting Network Communication Issues

• Verify physical and network layer connections

  • NTP on LAN

    • Verify communication between the client and the server using their MAC addresses properly

  • NTP outside LAN

    • Verify communication between clients and servers using Layer 3 IP addresses

• NTP packets are not being received

  • Indicates communication issue at Layer 1, 2, and 3, or a DNS server issue (using domain name)

• NTP received but not processed

  • Look at the network client or the NTP server to ensure they are operating the NTP service

• NTP process or service not acting on the NTP packets being received

  • Indicates network communication issues with other services, like HTTPS and network authentication processes

• Errors or packet loss in processed NTP packets

  • Can lead to time synchronization loss

    • High dispersion or delayed values

      • Indicate packets take too long to reach the client from the server, affecting time accuracy

  • Saturated links or buffering can delay NTP packets

  • Varying timestamps in NTP packets can disrupt synchronization

  • Resolution

    • Ensure network connections are not saturated and have adequate connectivity for timely NTP packet delivery