Splunk Cloud and Splunk Enterprise Quick Reference Guide

Splunk Cloud and Enterprise: Quick Reference Guide

Key Concepts

  • Events
    • A set of values associated with a timestamp.
    • Represents a single data entry, which can be single or multi-line.
    • Can be a text document, configuration file, stack trace, etc.
    • Example: 173.26.34.223 - - [01/Mar/2021:12:05:27 -0700] "GET /trade/app?action=logout HTTP/1.1" 200 2953
  • Transactions
    • Defined to search for and group conceptually related events that span a duration of time.
    • Can represent multi-step business activities, such as events within a single customer session on a retail website.
  • Metrics
    • A data point consisting of a timestamp and one or more measurements.
    • Can also contain dimensions.
    • Measurement: A metric name with a corresponding numeric value.
    • Dimensions: Provide additional information about measurements.
    • Sample Metric Data Point:
      • Timestamp: 08-05-2020 16:26:42.025 -0700
      • Measurement: metric_name:os.cpu.user=42.12, metric_name:max.size.kb=345
      • Dimensions: hq=us-west-1, group=queue, name=azd
    • Metrics and events can be searched and correlated but are stored in different types of indexes.
  • Host, Source, and Source Type
    • Host: The name of the physical or virtual device where an event originates. Used to find data from a specific device.
    • Source: The name of the file, directory, data stream, or other input from which an event originates.
    • Source Type: Classifies sources into well-known or user-defined formats. Common examples include HTTP web server logs and Windows event logs.
      • Events from different sources can share the same source type (e.g., /var/log/messages and UDP:514 might both be sourcetype=linux_syslog).
  • Fields
    • Searchable name and value pairings that differentiate events.
    • Not all events have the same fields or field values.
    • Enable tailored searches to retrieve specific events.
    • Extracted by Splunk software at index-time and search-time based on configuration files and user-defined patterns.
    • The Field Extractor tool can automatically generate and validate search-time field extractions using regular expressions or delimiters (spaces, commas, etc.).
  • Tags
    • A knowledge object for searching events based on particular field values.
    • Can be assigned to any field/value combination, including event types, hosts, sources, and source types.
    • Used to group related field values or to give descriptive names to abstract values like IP addresses or ID numbers.
  • Index-Time and Search-Time
    • Index-Time Processing:
      • Data is read from a source on a host.
      • Classified into a source type.
      • Timestamps are extracted.
      • Data is parsed into individual events, and line-breaking rules are applied.
      • Each event is written to an index on disk for later retrieval.
    • Search-Time Processing:
      • Occurs when a search starts; indexed events are retrieved from disk.
      • Fields are extracted from the raw text of the event.
  • Indexes
    • Where Splunk stores parsed and processed event data.
    • Data is parsed, timestamped, line-breaking rules are applied, and events are stored here.
    • New indexes can be created for different inputs.
    • By default, data is stored in the "main" index.
    • Events are retrieved from one or more indexes during a search.

Core Features

  • Search
    • The primary interface for navigating data in Splunk.
    • Allows retrieving events, calculating metrics with statistical commands, generating reports, identifying patterns, predicting trends, and more.
    • Events are transformed using the Splunk Search Process Language (SPL™).
    • Searches can be saved as reports and used to power dashboards.
  • Reports
    • Saved searches.
    • Can be run ad hoc, scheduled at regular intervals, or set to generate alerts based on specific conditions.
    • Can be added to dashboards as panels.
  • Dashboards
    • Composed of panels containing modules like search boxes, fields, and data visualizations.
    • Connect to saved searches.
    • Display results from completed searches and real-time searches.
  • Alerts
    • Triggered when search results meet specific conditions.
    • Configurable for historical and real-time searches.
    • Can trigger actions like sending emails or posting to web resources.

Additional Features

  • Datasets
    • Allows creating and managing different types of datasets: lookups, data models, and table datasets.
    • Table Datasets: Curated collections of event data designed for specific business purposes.
      • Can be defined and maintained using Table Views, a tool that simplifies complex search commands into UI editor interactions.
  • Data Models
    • Hierarchically organized collections of datasets.
    • Can be referenced entirely or selectively in searches.
    • Data Model Acceleration: Applied to data models for significant search performance gains, making them ideal for dashboard panels and essential on-demand reports.
  • Apps
    • Collections of configurations, knowledge objects, and custom views/dashboards.
    • Extend the Splunk environment for specific organizational teams (e.g., Unix/Windows admins, network security, web managers, business analysts).
    • A single Splunk Enterprise or Cloud installation can run multiple apps concurrently.
  • Distributed Search
    • Scales a Splunk deployment by separating the search management/presentation layer from the indexing/search retrieval layer.
    • Facilitates horizontal scaling for enhanced performance, controls access to indexed data, and manages geographically dispersed data.

System Components

  • Forwarders
    • A Splunk instance that sends data to another Splunk instance.
  • Indexer
    • A Splunk instance that indexes data.
    • Transforms raw data into events and stores them in an index.
    • Also searches indexed data in response to search requests.
    • Search Peers: Indexers that fulfill search requests from the search head.
  • Search Head
    • In a distributed search environment, this Splunk instance directs search requests to search peers and merges the results for the user.
    • If an instance solely performs searching (without indexing), it is called a dedicated search head.

Search Processing Language (SPL)

  • Syntax: A search is a series of commands and arguments.
    • Commands are chained together using a pipe | character, where the output of one command feeds into the next.
    • General Format: search | command1 arguments1 | command2 arguments2 | ...
    • The initial search command to retrieve events from the index is often implied.
  • Search Requests: Written with:
    • Keywords
    • Quoted phrases
    • Boolean expressions (AND, OR, NOT, XOR)
    • Wildcards (*)
    • Field name/value pairs (field=value)
    • Comparison expressions (field>value)
    • The AND operator is implied between search terms.
  • Example Search: sourcetype=access_combined error | top 5 uri
    • Retrieves indexed web activity events containing the term "error".
    • For those events, it returns the top 55 most common URI values.
  • Role of Search Commands:
    • Filter unwanted events (remove rows).
    • Extract more information (add columns).
    • Calculate values.
    • Transform and statistically analyze indexed data.
    • Search results are dynamically created tables: each indexed event is a row, and field values are columns.

Time Modifiers

  • Specify a time range for events using earliest and latest search modifiers.
  • Syntax: [+|-]<integer><unit>@<snap_time_unit>
    • integer: Numerical value.
    • unit: s (seconds), m (minutes), h (hours), d (days), w (weeks), mon (months), y (years).
    • @<snap_time_unit>: Optional; rounds the time down to the beginning of the specified unit.
      • Example: If it's 11:59:00 and you snap to hours (@h), the time becomes 11:00:00.
      • Can snap to specific days of the week: @w0 for Sunday, @w1 for Monday, etc.
  • Example Search: error earliest=-1d@d latest=-h@h
    • Retrieves events containing "error" that occurred yesterday (snapped to 00:00:00 of that day).
    • Continues through to the most recent full hour of today (snapped on the hour).

Subsearches

  • A subsearch runs independently and returns its results to the parent command as argument values.
  • Executed first and enclosed in square brackets [].
  • Example: sourcetype=syslog [ search login error | return 1 user ]
    • The subsearch search login error | return 1 user finds the user who had the last login error.
    • The parent search then finds all syslog events from that specific user.

Optimizing Searches

  • Limit Data: Pull the absolute minimum data off disk.
  • Filter Early: Apply filters as early as possible in the search pipeline to process less data.
  • Partition Data: Use separate indexes for different data types if cross-data searches are rare (e.g., web data in one index, firewall data in another).
  • Time Range: Limit to only what is needed (e.g., earliest=-1h instead of earliest=-1w or earliest=-1d).
  • Specificity: Search as specifically as possible (e.g., fatal_error instead of *error*).
  • Post-Processing: Use post-processing searches in dashboards.
  • Acceleration Features: Utilize summary indexing, report acceleration, and data model acceleration.

Machine Learning Capabilities

  • Integrated across Splunk's product portfolio and embedded in solutions via the Splunk Machine Learning Toolkit.

SPL2

  • A newer version of SPL used in several Splunk products.
  • Designed to be easier to use, removes infrequently used commands, and improves command syntax consistency.

Common Search Commands

  • chart/timechart: Returns results in tabular output for time-series charting.
  • dedup: Removes subsequent results that match a specified criterion.
  • eval: Calculates an expression and puts the result into a field.
  • fields: Removes fields from search results.
  • head/tail: Returns the first/last NN results.
  • lookup: Adds field values from an external source (lookup table).
  • rename: Renames a field (supports wildcards for multiple fields).
  • rex: Specifies regular expression named groups to extract fields.
  • search: Filters results to those that match the search expression.
  • sort: Sorts search results by specified fields.
  • stats: Provides statistics, optionally grouped by fields.
  • mstats: Similar to stats, but used for metrics instead of events.
  • table: Specifies fields to keep in the result set, retaining data in tabular format.
  • top/rare: Displays the most/least common values of a field.
  • transaction: Groups search results into transactions.
  • where: Filters search results using eval expressions, often for comparing two different fields.

Common Eval Functions

The eval command calculates an expression and places the resulting value into a field. It supports basic arithmetic (+ - * / %), string concatenation (.), and Boolean operations (AND, OR, NOT, XOR, <, >, <=, >=, !=, =, ==, LIKE).

  • abs(X): Returns the absolute value of X. Example: abs(number).
  • **`case(X,