T7b

Network Monitoring Fundamentals

• Relevance: Network monitoring is applicable nearly everywhere there's a network link (Ethernet, WiFi, etc.).

• OSI Model Layers: Monitoring spans across the Application, Transport, Network, Session, Presentation, and Physical layers.

• Data Representation: This involves messages, packets, frames, bits, and segments.

Network Monitoring: Measurement and Monitoring

• Core Objectives:

  • Measuring network activity.

  • Monitoring network performance.

  • Identifying what to measure and how to measure it.

• Key Questions to Answer:

  • Network Utilization: How busy is the network?

  • Congestion: Is there congestion, and where?

  • Errors: Are there errors occurring?

  • Hardware/Software Status: Is the hardware/software functioning correctly?

  • Bugs: Is there a bug in the network?

  • Changes: Has something changed for the worse or better?

  • Application Packet Delivery: Are applications receiving the correct packets?

  • Routing Behavior: Is routing functioning as expected?

Speed Testing

• Tools: Various tools like speedtest.net are used.

• Focus: Measures the performance of a device.

• Considerations:

  • Testing endpoint.

  • Testing time.

  • Environmental conditions (weather, distance, interference, topography, construction).

  • Number of users.

  • User behavior.

Performance Over Time

• Key Metrics:

  • Current, average, and maximum bits per second.

  • Inbound vs. outbound traffic.

• Applications:

  • Capacity planning.

  • Outage detection.

  • Pattern recognition.

Traffic Analysis and System Information

• Example System: MikroTik CCR GW in Jauher NOC.

• Maintainer: aacable@hotmail.com.

• Description: WAN1 + WAN2 + WAN3 Combined.

• Interface: WAN1+WAN2+WAN3.

• Maximum Speed: xxxx Mbits/s.

• Captured Data:

  • Maximum, average, and current bits per second.

Traffic Load

• Traffic Load Ranges: 1-10%, 10-25%, 25-40%, 40-55%, 55-70%, 70-85%, 85-100%.

Performance Over Space

• Network Status: Represented with a Tropicalex Network map.

• Visualization: Displays traffic load and network status geographically.

• Technologies Utilized

  • Cisco Catalyst 2960 Series Switches

Network Feedback Mechanisms

• ECN: Explicit Congestion Notification.

• ICMP: Internet Control Message Protocol (passive and active).

  • Error/alerts; ping; traceroute.

• TCP Acknowledgements.

• Application-level measures.

• Challenges:

  • No unified view.

  • No aggregated view in space or time.

• Data Sources: The devices on the network.

  • Traditional methods: Telnet/HTTP into each device.

  • Alternative: SNMP.

Administrative Domains

• Interior Domain (Your Network):

  • Authority: You have full control.

  • Data Access: Can obtain information from all devices.

  • Methods: Deploy software on each device for probing, measuring, and scanning.

• Exterior Domain (Beyond Your Network):

  • Authority: Limited or no authority.

  • Access: Limited access, possibly through contracts.

  • Methods: Request third parties to install software for data sharing.

Simple Network Management Protocol (SNMP) Design Requirements

• Reach: Universal accessibility.

  • Devices: Compatibility with switches, routers, access points, printers, servers, cameras.

  • Constraints: Support for small, simple, hard-to-reach, or outdated devices.

• Lightweight: Minimal device interference.

• Operability Under Stress: Ability to identify struggling/failing components.

• Scalability: Accommodate a large number of unknown device parameters.

  • Global naming; delegated; vendor-independent; extensible.

• Functionality: Both query/response and command/control capabilities.

• Security: Basic security features.

SNMP Framework

• Application Framework: For managing/monitoring network resources.

• Components:

  • SNMP protocol.

  • SNMP agents.

  • SNMP managers.

  • Management Information Bases (MIBs).

SNMP Components

• Agent: Software on the equipment.

  • Function: Maintains configuration and current state in a database.

  • Proxies: Agents that communicate with non-SNMP devices.

• Manager: Application that contacts an agent.

  • Function: Queries or modifies the database at the agent.

  • Part of Network Management Systems (NMS).

• Management Information Bases (MIBs): Describes the database.

  • Examples: MIB, MIB-II (RFC 1213), and many vendor specific MIBs.

  • Structure of Management Information (SMI): Defines sets of related objects in a MIB.

• SNMP protocol: Versions SNMPv1, v2, and v3.

SNMP Agent Information Design

• Lightweight Design:

  • No rate calculations.

  • No absolute clocks.

  • No history.

• Elements:

  • Counters and gauges.

  • Time since start-up.

  • Strings, Identifiers.

• Timeticks: Time in 1/100ths of a second.

• Functionality: Command/control through variable setting.

• Communication: A mix of query/responses and alerts.

SNMP Protocol

• Messages:

  • GetRequest: Retrieves information.

  • GetNextRequest: Retrieves the next object.

  • SetRequest: Modifies an object.

  • GetResponse: Response to a request.

  • Trap: Notification of an event.

• Architecture:

  • SNMP Management Station.

  • Management Application.

  • SNMP Agent.

  • Managed Resources.

  • SNMP Managed Objects.

• Transport: Uses UDP over IP.

SNMP Protocol Communication

• Ports:

  • Agent: Passive open on port 161 (Server).

  • Manager: Passive open on port 162 (Client).

• Message Exchange:

  • Manager sends request to Agent on port 161.

  • Agent sends response to Manager on port 162.

  • Agent sends trap message to Manager on port 162.

SNMP Proxies

• Function: Acts as an intermediary between a Management Station and a Proxied Device.

• Components:

  • Manager Process.

  • Proxy Agent.

  • Mapping Function.

  • Agent Process.

• Protocol: SNMP over UDP/IP.

SNMP Messages

• Connectionless: SNMP/UDP is connectionless and uses a request ID to maintain a session.

• PDUs: SNMP messages are protocol data units.

• Capabilities (SNMPv1):

  • Get: Retrieves the value of an object from an agent.

  • Set: Modifies the value of an object on an agent.

  • Notify: Informs a manager of an event at the agent.

SNMP(v1) Protocol Operations

• On-demand:

  • Get-request: Request the values of one or more objects.

  • Get-next-request: Requests the value of the next object.

  • Set-request: Modify the value of one or more objects.

  • Get-response: Agent response to a request.

• Triggered:

  • Trap: A notification from an agent to a manager about a specific event.

SNMP Traps

• Asynchronous: Traps are sent asynchronously by an agent to a manager.

• Core traps:

  • linkDown: An interface went down.

  • linkUp: An interface came up.

  • coldStart: Unexpected restart (system crash).

  • warmStart: Expected restart (manual reboot).

  • AuthenticationFailure: Somebody tried to query, but failed authentication.

  • egpNeighbourLoss: Link is up but the neighbor is gone.

• Additional traps: Approximately 232 vendor-specific traps.

Format of SNMP (v1/v2) Packets

• Components:

  • Version.

  • Community.

  • SNMP PDU (Protocol Data Unit).

    • PDU Type.

    • Request ID.

    • Error Status.

    • Error Index.

    • Object 1, Value 1.

    • Object 2, Value 2.

SNMP Community

• Definition: SNMPv1 defines communities to specify access to variable sets.

  • Access levels: read-write, read-only, none.

• Community Name: Included in each SNMP message.

  • Function: Acts like a password.

  • Security: Unencrypted.

• Typical Values:

  • Read-only: "Public".

  • Read-write: "Private".

• Enhancement: Agent/manager relationship based on the IP address of permitted managers, stored on the agent.

SNMP Versions

• Versions:

  • SNMPv1 (1990).

  • SNMPv2c (1996).

    • Adds "GetBulk" function.

    • Adds federated monitoring capabilities (manager to manager).

    • Adds TCP transport option.

    • Adds 64bit counters.

  • SNMPv3 (2002).

    • Addresses security.

• Adoption: All versions are still used; many agents/managers support all three.

SNMP Security

• SNMPv1: Uses community strings for authentication in plain text without encryption.

• SNMPv2: Security improvements were attempted but not fully realized; the "c" in SNMPv2c stands for "community".

• SNMPv3: Key security features:

  • Integrity: Ensures packet has not been tampered with.

  • Authentication: Ensures message is from a valid source.

  • Privacy: Ensures message cannot be read.

SNMPv3 Security Levels

• noAuthNoPriv: Authentication by matching a username.

• authNoPriv: Authentication with message digests.

• authPriv: Authentication with message digests and encryption.

SNMPv3 Architecture

• Manager Components:

  • NOTIFICATION RECEIVER

  • COMMAND GENERATOR

  • PDU DISPATCHER

  • COMMUNITY BASED SECURITY MODEL

  • USER BASED SECURITY MODEL

  • OTHER SECURITY MODEL

  • SECURITY SUBSYSTEM

  • SNMPv1, SNMPv2C, SNMPv3, OTHER MESSAGE PROCESSING SUBSYSTEM

  • MESSAGE DISPATCHER

  • TRANSPORT MAPPINGS

• Agent Components:

  • MANAGEMENT INFORMATION BASE

  • VIEW BASED ACCESS CONTROL

  • ACCESS CONTROL SUBSYSTEM

  • NOTIFICATION ORIGINATOR

  • COMMAND RESPONDER

  • PDU DISPATCHER

  • COMMUNITY BASED SECURITY MODEL

  • USER BASED SECURITY MODEL

  • OTHER SECURITY MODEL

  • SECURITY SUBSYSTEM

  • SNMPv1, SNMPv2C, SNMPv3, OTHER MESSAGE PROCESSING SUBSYSTEM

  • MESSAGE DISPATCHER

  • TRANSPORT MAPPINGS

Management Information Base (MIB)

• Storage: Values are stored in a Management Information Base (MIB).

• Organization: Collected under a Structure for Management Information (SMI).

• Characteristics: A sparse, voluntary/selective, infinitely-flexible database.

• Language: Written in ASN.1.

Abstract Syntax Notation One (ASN.1)

• Description: Formal description of data structures, message formats.

• Structure: Type, Length, Value (TLV).

• Basic Types:

  • BOOLEAN, INTEGER, OCTET STRING, BIT STRING, REAL.

  • ENUMERATED, CHARACTER STRING, OBJECT IDENTIFIER.

• Constructed Types:

  • SEQUENCE, SEQUENCE OF, CHOICE.

  • Arbitrary nesting of types and sub-types.

• Encoding Types: Multiple encoding types, but focusing on Basic.

ASN.1 Data Types

• INTEGER: A signed integer (4 bytes).

• Integer32: Same as INTEGER (4 bytes).

• Unsigned32: An unsigned integer (4 bytes).

• OCTET STRING: Variable-length byte string (up to $2^{16} - 1$ bytes).

• OBJECT IDENTIFIER: Variable-length object identifier.

• IPAddress: An IPv4 address (4 bytes).

• Counter32: Incrementing counter, wraps to zero after reaching $2^{32} - 1$ (4 bytes).

• Counter64: Incrementing counter (64-bit).

• Gauge32: Value between 0 and $2^{32} - 1$ (4 bytes).

• TimeTicks: Time in 1/100ths of a second (4 bytes).

• BITS: A string of bits (variable).

• Opaque: Uninterpreted data (variable).

IPv4 to IPv6 transition

• IPv4 Address: $IpAddress ::= OCTET STRING (SIZE (4))$

• Generic Address: $InetAddress ::= OCTET STRING (SIZE (0..255))$

• Address Types: $InetAddressType ::= INTEGER {unknown(0), ipv4(1), ipv6(2), ipv4z(3), ipv6z(4), dns(16)}$

Counters and Gauges

• Function: Counters and Gauges provide information about the current state.

  • Counter: Incremental value, such as packets on an interface, which can wrap.

  • Gauge: Represents a value between zero and a defined maximum, such as memory or disk space utilization.

• Agent Limitations: Agents do not store history or calculate rates/changes and have only a temporary clock representing time since the last boot.

• Manager Responsibilities: Managers must poll multiple times and make assumptions about the data.

  • Counter unchanged implies no change or a full wrap.

  • Gauge unchanged implies the state may or may not have changed between requests.

• MIB Design: May require multiple fields/types for related information.

ASN.1 OBJECT IDENTIFIER (OID)

• Purpose: Used to define and reference an information object at the international level.

• Structure: Managed at the international level, with internet OIDs under the iso org(3) dod(6) 1 branch.

• Uniqueness: Ensures global uniqueness.

OID Organization

• Hierarchy: OIDs are organized in a tree hierarchy, similar to DNS.

• Nodes: Each OID represents a node in the tree.

• Internet OIDs: Most internet objects are under 1.3.6.1.2.1.xyz.

• Private OIDs: Manufacturers can add product-specific objects under the <private> hierarchy 1.3.6.1.4.abc.

• SNMP Usage: SNMP uses OIDs for reference.

• MIB Mapping: MIBs map OIDs to human-readable forms, specifying their type, etc.

ASN.1 Examples

• Type Definitions:

  • $NumberOfStudents ::= INTEGER$

  • $PassOrFail ::= BOOLEAN$

  • $GradeType ::= ENUMERATED {A, B, C, D, E, F}$

  • $PointsScored ::= REAL$

  • $Image ::= BIT STRING$

  • $Data ::= OCTET STRING$

• Value Definitions and Assignments:

  • $studentsFridaySession NumberofStudents ::= 9$

  • $passCourse PassOrFail ::= TRUE$

• Combined Type/Value Definitions:

  • $StudentType ::= INTEGER { ugrad (0) ms (1) phd (2) }$

ASN.1 String Examples

• Access: "read-only", "read-write", "write-only", "not-accessible"

• Status: "mandatory", "current", "optional", "obsolete"

MIB "Object" Example

• ifNumber: Represents the number of network interfaces.

• Syntax: INTEGER.

• Access: read-only.

• Status: mandatory.

• Description: "The number of network interfaces (regardless of their current state) present on this system."

• OID: Located at { interfaces 1 }, which translates to 1.3.6.1.2.1.2.1.

MIB-2 Object Counting Packets

• ipForwDatagrams: Counts forwarded IP datagrams.

• Syntax: Counter.

• Access: read-only.

• Status: current.

• Description: "The number of input datagrams for which this entity was not their final IP destination…"

• Location: { ip 6 }, also known as 1.3.6.1.2.1.4.6.

Interface Information MIB-II

• ifMTU: Maximum packet size.

• ifSpeed: Interface speed in bits per second.

• ifPhysAddress: Physical address (e.g., MAC address).

• ifOperStatus: Operational status (Up(1), Down(2), Testing(3)).

• ifInErrors: Number of incoming packets discarded due to errors.

• ifInDiscards: Number of incoming packets discarded due to buffer overflow.

• ifOutQLen: Number of packets in the outbound queue.

• ifInUcastpkts: Number of incoming packets received.

Why Use OIDs?

• Global Uniqueness: OIDs ensure that each managed object is uniquely identified across all systems.

• Human-Readable Names: Provide descriptive names for each OID, which maps to positions in the OID tree.

• Table Representation: Used to create tables for data representation where ASN.1 lacks native table support.

Tables and GetNext

• Addressing: Each table cell is addressed using 1.3.6.1.2.x.y.z.abc.label.

• Indexing: Rows are indexed sequentially (e.g., interface number).

• Discovery: Managers don't need to know the number of rows beforehand.

• Operation:

  • Get("Interface.1.ipAddress") returns the IP address of interface 1.

  • Get-next("Interface.1.ipAddress") returns the IP address of interface 2.

• Order: Uses lexicographical order for OIDs.

Get-Bulk Request

• Problem: Repeated Get-next requests create significant traffic and require state management.

• Solution: Get-Bulk introduced in SNMPv2.

• Function: Retrieves every row and column in one request.

• Limitation: Error response "tooBig" if response exceeds 64kB UDP limit.

Performance Over Time Review

• Metrics Displayed

  • Incoming and Outgoing Traffic in Bits per Second

  • Maximum, Average, and Current Traffic Rates

• Tools Used

  • RRDTool, MRTG, Cacti, Nagios

• Example System

  • MikroTik CCR

Network Traffic Load and Status Visualization

• Traffic Load Representation: Displayed as percentages (e.g., 1-10%, 10-25%).

• Geographical Mapping: Network status overlaid on a map.

• Vendor Tools: Utilized for layout and drawing capabilities.

• Network Weathermap: Uses PHP and data from MRTG.

Monitoring Without SNMP

• Store-n-Forward: Capture on the device itself.

• Port Mirroring: Capture traffic on a mirror port using an extra device.

• Link Snooping/Sniffing: Capture from the cable directly.

• Insertion: Insert a router/LAN-switch and software (L2/L3 visibility).

• Sensors: Insert electrical/optical sensors.

SNMP Deployment Considerations

• SNMP in Wide Area Networks (WANs) is discouraged.

  • Security risks: SNMPv1/v2 agents should not be publicly accessible due to the risk of scanning.

• Administration: Requires asking for favors to other party for device/software installation.

Through the Looking Glass

• Description: Public interface provided by carriers/ISPs for remote access to their routers.

• Functionality: BGP queries, ping, traceroute.

• Examples:

  • Lumen

  • Telstra

Looking Glass Utilities

• Functionality:

  • BGP Route Lookup: View routing table entries for a specified prefix.

  • PING: View round trip time between a selected site and a specified IP address or hostname.

  • TRACEROUTE: View all enabled router hops between a selected site and a specified IP address or hostname.

Probing Beyond Your Network

• Strategy: Use dedicated probe points in peer communities.

• Measurements: Status, performance (bandwidth, latency, jitter), functionality (IPv6, multicast, etc.).

Network Beacons

• Function: Probes create live measurement tables.

• Implementation: Can be software or hardware.

• perfSONAR: An architecture for performance Service-Oriented Network monitoring (https://www.perfsonar.net).

Summary

• SNMP: Powerful for network management but not always available or suitable for WANs.

• Measuring: Essential for understanding link characteristics.

• Monitoring: Tracks performance over time and space.

• Snooping: Captures actual traffic (for legitimate performance analysis purposes only!).