SEC-201: Incident Detection and Digital Forensics - Comprehensive Study Notes

Incident Detection Fundamentals

Identification of malicious activities is a core pillar of cybersecurity operations. To effectively secure an environment, security professionals must address three foundational questions:

• How can we identify malicious activities?

• What kind of evidence is generated when an intruder breaches a network?

• Why is it essential to recognize these indicators within our environment?

Logs serve as the primary records of past events, providing the visibility and context required to answer these questions. By maintaining an archive of historical activity, organizations can bolster their security posture and protect digital assets.

Logs offer evidence value across several domains:

• Security operations and incident response.

• Threat hunting.

• Compliance and cyber resilience.

• Evaluating cyber maturity.

• The correlation of simultaneous and distinct events across distributed networks.

Anatomy of Logs and Log Entries

Logs are records of events within a system. They provide a detailed account of system behavior, capturing data such as user logins, file accesses, system errors, network connections, and modifications to data or system configurations.

While details vary by log type, a standard log entry typically includes:

• Timestamp: The exact time the event was logged.

• System/Application Name: The origin of the log entry.

• Event Type: The nature of the activity that occurred.

• Contextual Details: Identification of the user who initiated the event or the IP address of the source device.

Logs are designed to answer critical forensic questions:

• What happened?

• When did it happen?

• Where did it happen?

• Who is responsible?

• Were their actions successful?

• What was the result of their action?

Log Formats

Log formats are categorized based on their structure into three main types:

Semi-structured Logs

These contain a mix of structured and unstructured data, often featuring predictable components alongside free-form text. Examples include:

• Syslog Message Format: A widely adopted protocol for system and network logging.

• Windows Event Log (EVTX) Format: A proprietary Microsoft format for Windows systems.

Structured Logs

These follow strict, standardized formats that are ideal for automated parsing and analysis:

• Field Delimited Formats: Comma-Separated Values (CSV) and Tab-Separated Values (TSV).

• JavaScript Object Notation (JSON): Favored for readability and compatibility with modern programming languages.

• eXtensible Markup Language (XML): A flexible, customizable format used for standardized logging.

Unstructured Logs

These consist of free-form text, which is rich in context but difficult to parse systematically:

• Common Log Format (CLF): Used by default by Apache HTTP Server for client requests.

• Combined Log Format (Combined): An extension of CLF used by Nginx, which includes additional fields like the referrer and user agent.

Log Collection and Management

Log collection involves aggregating logs from servers, network devices, software, and databases. To ensure logs accurately represent a chronological sequence of events, maintaining time accuracy is critical.

• Time Synchronization: The Network Time Protocol (NTP) should be utilized to synchronize system clocks, ensuring the integrity of the log timeline.

Effective Log Management follows a specific lifecycle to ensure data is secure and retrievable:

• Storage: Choosing secure solutions based on retention periods and accessibility requirements.

• Organization: Classifying logs by source or type for easier retrieval.

• Backup: Performing regular backups to prevent data loss.

• Review: Periodically checking categorization and storage integrity.

Log Centralization and the Analysis Process

Centralization is essential for rapid incident response and in-depth analysis. A centralized system provides real-time detection, automatic notifications, and integration with incident management modules.

Centralization Process

1. Choose a System: Utilize platforms like the Elastic Stack (ELK) or Splunk.

2. Integrate Sources: Connect all log-generating entities to the central hub.

3. Set Up Monitoring: Implement tools for real-time alerts on specific events.

4. Integration: Ensure seamless connectivity with incident management protocols.

The Log Analysis Workflow

1. Data Sources: The origin systems/applications generating the logs.

2. Parsing: Breaking log data into manageable components to extract valuable info.

3. Normalization: Standardizing parsed data into a common format for comparison.

4. Sorting: Arranging data by time, source, type, or severity to identify trends.

5. Classification: Categorizing logs (e.g., by severity) to filter and focus on critical data.

6. Enrichment: Adding context, such as geographical data, user details, or threat intelligence.

7. Correlation: Linking related records to identify hidden relationships between events.

8. Visualization: Representing data as charts, graphs, or heat maps for easier interpretation.

9. Reporting: Summarizing data for stakeholders, management, or auditors.

Log Analysis Techniques

Analysts use various methods to derive insights from log data:

• Pattern Recognition: Identifying recurring sequences to detect normal versus unusual behavior.

• Anomaly Detection: Focusing on data points that deviate from expected patterns to spot malicious activity.

• Correlation Analysis: Understanding causation and dependencies between system components through related logs.

• Timeline Analysis: Analyzing logs over time to understand trends, seasonality, and load forecasting.

• Machine Learning and AI: Automating classification, enrichment, and predictive responses.

• Statistical Analysis: Using quantitative methods like regression analysis and hypothesis testing to validate assumptions.

Sources and Types of Logs

Common Log Types

• Application Logs: Status messages, errors, and warnings from specific apps.

• Audit Logs: Operational records required for regulatory compliance.

• Security Logs: Security events like logins, firewall activity, and permission changes.

• Server Logs: System, event, error, and access logs produced by servers.

• System Logs: Kernel activity, boot sequences, and hardware status.

• Network Logs: Traffic logs and connection data.

• Database Logs: Queries and updates within a database system.

• Web Server Logs: Processed requests and response codes.

Data Sources

• Network Components: Routers, switches, and plug-in devices.

• Operating Systems: Servers.

• Security Appliances: Firewalls, Anti-virus, IDS/IPS, DLP, and VPNs.

• Applications: Frameworks like .Net, Java, PHP, and Python.

• Mobile Devices.

• Virtualization/Cloud Components.

• Databases.

Windows Logging Specifics

Windows logs use the default extensions $.evt$ or $.evtx$. They can be extracted as $.xml$, $.csv$, or $.txt$.

File Locations

• Windows 2003 and earlier: C:\WINDOWS\system32\config

• Windows Vista and newer: C:\WINDOWS\System32\winevt\Logs

Log Components

• Event ID: A numeric identifier for the event type.

• Level (Severity): Information (success), Error (system issue), Warning (potential problem), Critical (significant issue), and Verbose (progress messages).

• Task Categories: Common examples include Process Creation, Service Creation, and Log Clear.

• Keywords: Standard attributes like Audit Success or Audit Failure.

Critical Windows Event IDs

• Account Management:

  • $4720$: User account creation.

  • $4722$: User account enabled.

  • $4723$: Attempt to change account password.

  • $4724$: Attempt to reset account password.

  • $4725$: Account disabled.

  • $4726$: Account removed.

• Logon/Logoff:

  • $4624$: Successful logon.

  • $4625$: Failed logon.

  • $4634$ and $4647$: Logoff.

  • $4779$: Session disconnect.

• Scheduled Tasks:

  • $4698$: task creation.

  • $4702$: task update.

  • $4699$: task deletion.

• Security:

  • $1100$: Logging service disabled.

  • $1102$: Log deletion.

  • $1116$: Malware detection.

Linux Logging Specifics

Linux systems typically store logs in $/var/log$. Format varies by service:

• Syslog: Cleartext ($.log$).

• Systemd/Journald: Binary format.

Common Linux Log Paths

• System Logs: $/var/log/syslog$

• Authentication: $/var/log/auth.log$

• Nginx Access: $/var/log/nginx/access.log$

• Nginx Error: $/var/log/nginx/error.log$

• Apache Access: $/var/log/apache2/access.log$

• Apache Error: $/var/log/apache2/error.log$

• MySQL Error: $/var/log/mysql/error.log$

• PostgreSQL: $/var/log/postgresql/postgresql-{version}-main.log$

• iptables (Firewall): $/var/log/iptables.log$

• Snort (IDS): $/var/log/snort/$

Log Analysis Tools

Linux Command Line Interface (CLI)

• cat: Displays full log content.

• tail: Views the end of files. Use the $-f$ option to follow logs in real-time. Displays last $10$ lines by default.

• wc (word count): Provides stats on lines, words, and characters.

• cut: Extracts specific columns based on delimiters.

• sort: Arranges entries chronologically or alphabetically.

• uniq: Removes adjacent duplicate lines from sorted input.

• grep: A powerful text search tool for pattern matching.

Advanced Tools

• CyberChef: Referred to as the "Cyber Swiss Army Knife," it features over $300$ operations for encoding, decoding, encryption, and log parsing.

• Sigma: An open-source tool using YAML syntax to describe log events. It is used to detect events, create searches, and identify threats through pattern matching.

• Yara: A YAML-formatted tool for identifying information based on binary and textual patterns (hexadecimal and strings). Originally for malware analysis, it is highly effective for log analysis.