Cybersecurity Home Lab Notes — Today

1. Fixed the Splunk Forwarding Issue

Problem

Splunk was showing:

The TCP output processor has paused the data flow

Investigation

Checked forwarding configuration:

sudo /opt/splunk/bin/splunk list forward-server

Found:

Configured but inactive forwards:
    :9997

This meant Splunk was trying to forward logs to a destination that didn't exist.

Root Cause

Found invalid configuration:

sudo cat /opt/splunk/etc/system/local/outputs.conf

Contents:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = :9997

Fix

Backed up and removed:

sudo cp /opt/splunk/etc/system/local/outputs.conf /opt/splunk/etc/system/local/outputs.conf.bak

sudo rm /opt/splunk/etc/system/local/outputs.conf

Restarted Splunk:

sudo /opt/splunk/bin/splunk restart

Verified:

sudo /opt/splunk/bin/splunk list forward-server

Result:

No forwarders configured

2. Verified Ubuntu Services

Apache

Verified Apache was running:

sudo systemctl status apache2

SSH

Verified SSH service:

sudo systemctl status ssh

Listening Ports

Checked listening services:

sudo ss -tulpn

Found:

22   SSH
80   Apache
8000 Splunk Web
8089 Splunk Management

3. Generated First Security Event

Attack From Kali

Executed:

ssh fakeuser@192.168.10.103

Entered invalid passwords multiple times.

Evidence on Ubuntu

Viewed logs:

sudo tail -20 /var/log/auth.log

Observed:

Invalid user fakeuser
Failed password
Authentication failure
Connection closed

4. First SOC Investigation

Attacker

192.168.10.104
(Kali)

Victim

192.168.10.103
(Ubuntu)

Activity

SSH brute-force attempt

Evidence

Invalid user
Failed password
Authentication failure
Connection closed

Severity

Low

5. Splunk Log Ingestion Troubleshooting

Problem

Searching:

index=* "Failed password"

Returned:

No results

Investigation

Found Splunk was not monitoring auth.log.

Checked:

grep -R "/var/log/auth.log" /opt/splunk/etc

Returned nothing.

Created Monitoring Input

Edited:

sudo nano /opt/splunk/etc/system/local/inputs.conf

Added:

[monitor:///var/log/auth.log]
disabled = 0
index = main
sourcetype = linux_secure

Verified

Checked:

sudo /opt/splunk/bin/splunk list monitor

Observed:

/var/log/auth.log

was being monitored.

6. Tested Splunk Ingestion

Created custom event:

logger "SOC-LAB-TEST-EVENT-12345"

Verified event existed:

sudo tail -20 /var/log/syslog | grep SOC-LAB

Output:

SOC-LAB-TEST-EVENT-12345

Added Syslog Monitoring

Updated:

[monitor:///var/log/syslog]
disabled = 0
index = main
sourcetype = syslog

Restarted Splunk:

sudo /opt/splunk/bin/splunk restart

Generated new event:

logger "SOC-LAB-TEST-EVENT-67890"

Success

Searched Splunk:

SOC-LAB-TEST-EVENT-67890

Result:

Event Found

This proved:

Ubuntu Logs
     ↓
Splunk Monitoring
     ↓
Splunk Indexing
     ↓
Splunk Search

WORKS.

7. Networking Status

Ubuntu

192.168.10.103

Kali

192.168.10.104

pfSense

192.168.10.1

Management VLAN

VLAN 10
192.168.10.0/24

8. Nmap Enumeration

From Kali:

nmap -sV 192.168.10.103

Discovered:

22/tcp   SSH
80/tcp   Apache
8000/tcp Splunk Web
8089/tcp Splunk Management

What You Learned Today

Linux

• 
  systemctl
  
  

• 
  ss
  
  

• 
  auth.log
  
  

• 
  syslog
  
  

• 
  logger
  
  

• 
  journal/syslog concepts
  
  

Networking

• 
  IP addressing
  
  

• 
  SSH communication
  
  

• 
  Service enumeration
  
  

• 
  Port scanning
  
  

Splunk

• 
  outputs.conf
  
  

• 
  inputs.conf
  
  

• 
  Forwarders
  
  

• 
  Indexes
  
  

• 
  Sourcetypes
  
  

• 
  Monitoring files
  
  

• 
  Log ingestion troubleshooting
  
  

Detection & Monitoring

• 
  Failed SSH logins
  
  

• 
  Security log analysis
  
  

• 
  Event generation
  
  

• 
  Event ingestion
  
  

• 
  Basic SOC workflow
  
  

Current Lab Architecture

Kali Linux (192.168.10.104)
        │
        │ Attacks / Nmap / SSH
        ▼
Ubuntu Server (192.168.10.103)
        │
        │ auth.log
        │ syslog
        ▼
Splunk Enterprise
        │
        ▼
Detection & Monitoring

pfSense (192.168.10.1)
        │
        ▼
UniFi Switch

Next Session Goals

1. 
   Get SSH failed login events searchable in Splunk.
   
   

2. 
   Create first Splunk detection search.
   
   

3. 
   Create first Splunk alert.
   
   

4. 
   Build dashboard for SSH activity.
   
   

5. 
   Test Nmap detection.
   
   

6. 
   Begin planning SOAR integration (Shuffle or TheHive/Cortex).
   
   

7. 
   Create automated response workflow using pfSense.