Network Monitoring and Intrusion Detection

Principles of Network Monitoring and Intrusion Detection

• Network monitoring and intrusion detection (IDS) serve as the fundamental first steps in security operations, facilitating the discovery of malicious events prior to planning and implementing a response.

• Malicious activity is rarely identifiable through a single, isolated event, such as a lone DNS request or connection to a website. These actions must be analyzed in context to determine if they constitute a threat.

• Detection is achieved through the correlation of events, which involves examining the order of events and the combination of multiple actions to identify identifiable patterns of behavior.

• A typical malicious pattern involves a progression: a workstation queries a suspicious domain, which then spawns a word processing application, which subsequently launches an obfuscated script. Individually, these actions might appear benign or only slightly suspicious, but in combination, they indicate a clear security incident.

• The objective of network monitoring is to collect and analyze evidence from network communications to understand activity, detect suspicious behavior, and support forensic investigations.

• Monitoring identifies critical details, including the parties involved (victim machines versus remote attackers), affected internal systems, the protocols utilized ($DNS$, $TLS$, etc.), and any files or payloads transmitted.

Taxonomy of Network Security Monitoring Data

Traditional network security monitoring relies on four primary types of data to provide a comprehensive view of network activity:

• Full Content Data: This consists of raw packet captures ($PCAP$), similar to those generated by Wireshark. While it provides every detail of a communication, it is extremely expensive to maintain constantly and can rapidly fill storage logs, even on single machines.

• Transaction Data: This is a summarized version of full content data, providing a high-level view of lower-level packet activity. It is easier to analyze and saves significant storage space. Tools like Zeke provide this record of events.

• Extracted Content: This refers to files or specific data fragments pulled from the network traffic, such as malware payloads delivered to a host or sensitive files being exfiltrated to a remote location.

• Alert Data: These are automated detection mechanisms that signify when traffic matches a predefined suspicious pattern. Alerts reduce the massive volume of total activity to a small subset that human analysts can feasibly investigate to determine if a threat is a false positive or a legitimate incident.

Encryption and Visibility Constraints

• Encryption is widely adopted for privacy and security, but it limits the visibility of network security analysts. By default, encrypted data remains opaque unless the $TLS$ endpoint is terminated at a middlebox (such as a router) before being forwarded.

• Despite encryption, critical metadata remains visible and useful for monitoring:

  • Source and Destination $IP$ addresses.

  • Source and Destination Ports.

  • Protocols used.

  • Certificate metadata (in $TLS\,1.3$, though some fields are increasingly encrypted).

  • Hostnames involved in the connection establishment.

• Information specifically hidden by encryption includes $HTTP$ requests, headers, cookies, and the actual content of downloaded files.

• In a $TLS$ handshake, the initial exchange is unencrypted, revealing identity and negotiation parameters, but communications become encrypted immediately after the session key is established.

Intrusion Detection Methodologies

Detection logic generally falls into two primary categories:

• Signature-Based Detection: This method looks for well-defined patterns of known malicious activity. Tools like Suricata use signatures to compare incoming packets against a database of known threats. If a match is found, an alert is generated.

• Anomaly-Based Detection: This approach focuses on defining a baseline of "normal" behavior for a specific workstation or user. Any activity that deviates significantly from this baseline is flagged as potentially malicious.

• Sensors can operate in two modes:

  • Passive Sensors: These collect and reflect data as it passes through the network without interfering with the traffic flow.

  • Inline Sensors: These act as gatekeepers, taking in packets and possessing the ability to block traffic deemed malicious. This functions as a second layer of defense after the firewall.

Practical Implementation Tools

• $TCPDump$: A standard tool used for capturing every packet that leaves or enters an endpoint device, creating raw record files for subsequent analysis.

• $Nmap$: A service discovery tool used to scan servers for open ports and identify the specific services and versions running on those ports. Attackers utilize it to find vulnerabilities; analysts use it to simulate and understand discovery-phase attacks.

• Suricata: A standard enterprise-level tool used for automated pattern detection and alert generation. It can be integrated into Security Information and Event Management ($SIEM$) systems like Splunk. Suricata analyzes logs and traffic against specific rules.

• Zeke (formerly Bro): A passive network analyzer that organizes the "undifferentiated mess" of raw packets into structured logic logs based on protocol. It generates specific files such as $dns.log$, $tls.log$, $http.log$, and $con.log$ (connection log).

Zeke Connection Log ($con.log$) Interpretation

Zeke summarizes network transactions using specific history and state codes to provide a high-level timeline of events:

• $S0$: An initial connection attempt was made (a $SYN$ packet was sent), but no reply followed. This often indicates a connection blocked by a firewall or a non-responsive host.

• History code $S$: Indicates a single $SYN$ packet was observed.

• $RSTO$: The connection was successfully established, but the originator (the client) aborted the connection shortly after.

• History code $SAr$: Indicates a $SYN$ packet was sent, a $SYN-ACK$ was received, and then a Reset ($r$) was triggered by the originator.

• $SF$: Indicates a normal connection setup and termination through the standard $FIN$ handshake.

• Service Discovery Analysis: In an $Nmap$ scan scenario, an analyst might see a sequence of $S0$ states for blocked ports, followed by $RSTO$ or $SF$ states for open ports where the attacker established a connection to grab the service version (e.g., discovering $OpenSSH\,v7.0$ running on a $Debian$ server) and then disconnected.

Industry Perspectives and Security Operations

• False Positives: A significant challenge for Security Operation Center ($SOC$) analysts is the high volume of false positives generated by $IDS$ and $IPS$ (Intrusion Prevention System) tools. Many alerts, such as failed database logins, often turn out to be legitimate scheduled activity rather than brute force attacks.

• Prioritization: Analysts prioritize signature-based alerts over anomaly-based ones, especially those involving Command and Control ($C2$) connections to known malicious domains or malware signatures, as these have a higher probability of being true positives.

• Endpoint vs. Network Detection:

  • $EDR$ (Endpoint Detection and Response): Monitors internal processes and activities directly on the host machine.

  • $NDR$ (Network Detection and Response): Monitors high-level malware activity and traffic flow across the network.

  • A robust defense strategy requires both $EDR$ and $NDR$ to be used complementarily for full attack chain visibility.

• Advanced Persistent Threats ($APT$): These are highly skilled adversaries who prioritize "laying low" in an environment. They move laterally across networks very silently to avoid triggering detection mechanisms. Detecting $APTs$ requires advanced threat intelligence and active "threat hunting" rather than relying solely on automated alerts.

• $SIEM$ (Security Information and Event Management): Systems like Splunk act as a central repository where logs from various tools (Suricata, Zeke, $EDR$) are consolidated for high-level correlation and investigation.