Live Workshop icon

July 16: Financial Services Roundtable

How-To Guide

Use log management as the foundation of the security stack

Contents
How-To Guide

Use log management as the foundation of the security stack

With the increase in endpoints, distributed systems, and machine data comes new opportunities for security breaches from inside and outside the organization. Organizations are responsible for protecting their infrastructure, their data, and the private data of their customers. Security teams are responsible for responding when incidents occur, and finding the root-cause to prevent it from happening again.

The toughest security incidents are unforeseen, and often happen where monitoring isn’t in place. How can organizations prepare for the unknown?

With comprehensive and real-time log monitoring of apps, systems, and endpoint data, organizations can minimize recovery time by making it easier to search all relevant data to find the root cause of an incident. This type of visibility gives security teams the continuous insights needed for immediate actions and data-driven responses to strengthen the security across systems, prevent downtime, and protect against attacks.

Modern log management has the ability to collect, analyze, search, and store hundreds of terabytes a day of log, metrics, traces, and events.

Security Information and Event Management (SIEM) platforms are powerful tools for monitoring systems for threats. Companies use SIEMs to protect their company’s data, and to show proof that they are meeting compliance requirements. However, many organizations simply track pre-selected data, leaving blind spots in their monitoring. Due to the ways SIEMs index data, increasing file sizes and processing times, they often make it prohibitively expensive to ingest and store all the data they should.

To solve these challenges, many organizations choose to enhance the capabilities of their SIEM by leveraging log management. This can enhance overall security, save time, and ease budget constraints.

This guide outlines how to center a security stack around log management as a way to increase an organization’s security profile without overloading the limits of the security team or the security solutions already in place. We share the following 5 steps as a way to get started, with more detailed information later in the page.

5 Steps to use modern log management for security

Determine security use cases

Prioritize the types of threats that the organization needs to address and prevent, based on the resources available and risk to operations.

Read more about each step

Section icon for: Integrating log management into security stackIntegrating log management into security stack

Gartner shares insight into how to think about using log management as part of the security stack in Use Central Log Management for Security Operations Use Cases (Gartner subscription required).

“Central log management is an important, but often undervalued, tool for an organization’s threat management capabilities and compliance requirements. Security and risk management leaders can benefit from the adoption of a log management tool for multiple security operations use cases.

“Modern security operations center activities require access to log data from a variety of sources that may be too expensive to consume in a SIEM solution. However, analysts investigating events may need access to this additional data for context and correlation, and threat hunters need access to a broad scope of data to do their job.” 1

Gartner describes how modern log management is used for security. They provide the following use cases for log management emerging from using Central Log Management (CLM), in addition to what was its primary purpose, IT Operations.

Organizations are using log management to track down root causes and decrease mean time to resolution for security use cases. In our opinion, log management boosts the abilities of a security response team by providing:

  • Basic threat monitoring with alerts and dashboards

  • Threat investigations to search correlated events for the root cause of incidents

  • Threat hunting to search historical data to discover unmonitored threats

  • Complementing SIEM and MSS/MDR (Managed Security Services/Managed Detection and Response) being used by the organization.

Section icon for: Log management creates a strong foundation for security dataLog management creates a strong foundation for security data

By collecting all appropriate logs and event data, log management provides a strong foundation to support all aspects of data security. With data stored from all endpoints, networks, applications, users, and processes, threats can be identified in seconds, and security incidents can be investigated months into the past.

Main Features

  • Collect and normalize logs and events

  • Dashboards and alerts

  • Search across all data

  • Data retention and long-term storage

Main Features

  • Threat detection

  • Threat dashboards and alerts

  • Compliance

  • Reporting

Main Features

  • Comprehensive threat hunting

  • Detailed root cause

  • Dashboards with app and infrastructure status

  • Longer security data retention

Modern log management as security platform

Building a security platform using log management provides a cost-efficient way to collect data and events for security alerts, investigations, and compliance. Modern log management technology delivers real-time dashboards and alerts, and offers security teams a streamlined way to search deeper into the past. Organizations large and small use log management to collect logs and events, and use the capabilities of log management to build out the features they need for data security.

Log management

A strong foundation to manage data security

Modern log management adds value to an installed SIEM

For organizations with a SIEM platform, modern log management can run alongside a SIEM and provide additional business value. This approach keeps the benefits provided by the existing SIEM tool, such as threat detection, incident response, and reporting. The organization would then use log management to ingest additional data sources. It may be possible to reduce the logs being ingested by the SIEM by collecting them with log management. Because log management platforms are optimized for fast searching and efficient storage, it enhances the organization’s ability to detect, investigate, and manage security data. By reducing the amount of data being managed and retained by the SIEM, this solution can significantly reduce the total cost of managing and storing log and event data.

Together for comprehensive coverage and optimized storage

Section icon for: Challenges managing SIEMsChallenges managing SIEMs

The implementation of each of these use cases comes with different challenges.

Excessive cost of SIEM log collection

Most SIEM tools are built on data platforms with substantial costs of ingest and retention. This limits the data that’s available for threat detection, alerts, and investigation, and puts SIEMs out of reach for many mid-sized and small enterprise businesses. Log management centralizes all logs and data from across the system, improving exploration and longer retention to meet compliance requirements.

Isolated views of security data

In many organizations, security solutions end up becoming an isolated solution used solely by security operations. As the security team grows, they can drift further from the developer or DevOps team, and lose visibility into the entire attack surface. By using the SIEM together with the central log management platform, both teams can get closer to the data being streamed into both solutions.

Limited data sources

Some SIEM solutions don’t have the ability to collect or manage logs outside of what the solution provides, limiting the scope of the search to what they define is adequate.

Limits on data retention

SIEMs may limit the amount of retention, or charge unreasonable amounts for storage of data beyond 30 or 90 days. Most managed security service providers (MSSPs) store 90 days of data, and may take hours to retrieve it. With a local log management platform, data can be retained at a much lower cost, making it possible to retain data for much longer.

Inflexible license costs

It’s often difficult for SIEM users to find a budgetary sweet spot. They’re often either underinvested in their SIEM and stuck without enough budget to monitor everything in their system, or they’re over-licensed and stuck paying higher costs for services they don’t use. For these SIEM users, modern log management provides a third option: they can scale back their investment in their SIEM and accomplish everything they need to with robust log management at lower license and maintenance costs.

Limited data throughput

SIEMs aren’t optimized for data throughput like modern log management, so they may actively discourage using bandwidth. They may charge based on the number of users, putting pressure on how users interact with the data. Modern log management makes data more freely available around the company, providing additional business value and better insights from data.

Limited security data visibility

A visibility challenge occurs when a suspected incident is detected but the contextual data isn’t available. The security administrator has to go looking outside of the SIEM to understand the scope of the compromise. And often, the data needed is gone because the logs have been rotated. SIEMs on their own provide a limited picture of the system. In order to reduce the load on their already resource-intensive pattern analysis, they use a curated set of log data — so they aren’t seeing the whole picture. Without a robust set of logs and event data, security administrators may become embroiled in what may be a several-week process of manually accessing and reviewing data sources.

Delayed data prevents real-time detection

Many security systems rely on data that is indexed and stored before it’s available for alerts, delaying the time to detection and resolution. Modern log management streams real-time data without indexes, updating alerts in real time, and allowing investigation and searching the moment an incident occurs.

Section icon for: Choose the right log management solutionChoose the right log management solution

For the best results, look for a modern log management solution optimized for speed and efficiency. Look for these hallmarks to find the best high-throughput, low-cost system.

Checklist

  • Architecture for speed, efficiency, and flexibility

  • Affordable license fees that scale predictably as data requirements grow

  • Capacity to ingest and store all data required

  • Streaming data ingest in real time

  • Fast search with near-zero latency from ingest to being searchable

  • Easy-to-use free-text search

  • Data enrichment to augment raw data, including joins from multiple data sources

  • Dashboards updated in real time

  • Flexible visualization capabilities

  • Data compression for efficient storage and data transfer

  • Long-term retention and storage using inexpensive cloud storage

  • Resilient design that doesn’t require extensive ongoing maintenance

  • Enterprise-level security

  • Self-hosted or SaaS

  • Self-hosted or SaaS

5 steps to use log management to monitor and react to security issue

Some companies may choose to use log management without purchasing a SIEM. Modern log management provides a powerful platform for setting up queries and alerts from the same sources used by SIEM tools. In fact, most SIEM tools started as log management tools where data sources, queries, visualizations, and alerts were added over time.

This approach leverages the full power and speed of modern advances in log management platforms, and may pay back the initial time investment. However, it does require security expertise and developer hours to put the system in place. Start with one use case, optimize it, and build the system for additional use cases.

Before building out a security system, it’s important to start by understanding the use cases the system must address. There are a few places to start when deciding which use cases to prioritize.

Compliance

Ensuring that data is handled correctly and securely is regulated by governments. In many cases, there are specific requirements for creating policies for data governance, keeping data secure, protecting consumer data, and retaining records of compliance for auditing purposes. Log management systems can help collect system, personal, and security logs, and store them to show compliance.

External threats

Of course there are countless ways for an external entity to infiltrate an organization and cause harm. Here are a few use cases that should be considered when designing a system to protect data.

Insider threats

Insider threats can be hard to find since it is often conducted by employees with access that seems legitimate. Make sure security use cases take these types of threats into account.

Adversary tactics and techniques and actions to take

For a more detailed list of possible threats integrated with cyber threat intelligence, MITRE publishes a list of tactics used by advisories to infiltrate or harm organizations and steal or destroy their data. They outline tactics, techniques, and information for categories that are part of the cyber kill chain: Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, Collection, Command and Control, Exfiltration, and Impact.

MITRE ATT&CK can help determine where to start

MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community. ATT&CK is open and available to any person or organization for use at no charge. For more information, see Getting Started with ATT&CK.

The ATT&CK Matrix maps the relationship between threats, tactics, and techniques. It is probably the most widely recognizable aspect of ATT&CK because it is commonly used to show things like defensive coverage of an environment, detection capabilities in security products, and results of an incident or red team engagement.

With security use cases prioritized, it’s time to begin collecting data. There are places to start that will become the foundation of security operations.

Threat intelligence is knowledge about threats based on evidence. It contains actionable information on how threats occur, the mechanisms used, context and implications, and specific advice to prevent harm. Threat intelligence can be used to help make decisions about preventing or responding to each threat.

There are several places to get updated threat intelligence, many of which are provided as a community service. Some of these are databases of indicators of compromise, which can be used to alert administrators to possible threats.

Threat intelligence can be used to enhance log data. For example, by looking at new IP addresses and comparing them against a list of bad actors, you can set up an alert when an identified IP address accesses the system.

Open source threat intelligence

There are dozens of open-source options to find information about threats, including these:

  • AT&T Alien Labs Open Threat Exchange. OTX Pulse provides a summary of threats, a view into the software targeted, and the related indicators of compromise (IOC) that can be used to detect the threats. IOCs include IP addresses, domains hostnames (subdomains), email, URL, URI, file hashes (MD5, SHA1, SHA256, PEHASH, IMPHASH), CIDR rules, file paths, MUTEX name, and CVE number.

  • Cisco Talos Intelligence IP Reputation Portal has a range of tools designed for the security investigator including IP Reputation.

  • RiskIQ Community Edition is a portal set up for the community to research security issues using RiskIQ’s extensive data..

  • Shodan IO scans for a range of Internet devices, breaking them down into industry categories. It is one of the first to focus on IoT devices that are vulnerable or have been violated.

  • Symantec’s Security Center SPAM Query Tool maintains a list of malware and vulnerabilities, and has an IP check tool for known spammers.

  • URLhaus – URLhaus is a project operated by abuse.ch. The purpose of the project is to collect, track, and share malware URLs, helping network administrators and security analysts to protect their network and customers from cyber threats.

  • Zeek is a flexible, open-source network monitoring tool powered by defenders. Zeek interprets what it sees and creates compact, high-fidelity transaction logs, file content, and fully customized output suitable for review with a log management tool or SIEM.

Lists of open source threat intelligence

These sites maintain a list of threat intelligence sources.

To make the data being collected in the log management system more useful for security, set up a dashboard to analyze information about each use case. Use queries to create visualizations that illustrate the results that support the prioritized use case. Create appropriate alerts to notify security personnel when high-profile threat indicators occur that need investigating.

Once a use case has been correctly set up, begin to work on the next use case. As the dashboard is used for analysis, adjust it to make it work for the organization. Optimize the data being collected and the queries driving the visualizations to make achieve the best results for the use case. When the desired results are being successfully achieved for the use case, move to the next prioritized use case.

Continually adjust the dashboard to make it work for the organization. Make sure it illustrates the most important information, and that it helps the business prioritize action. Make sure the data being reported provides value, and that it answers relevant questions for the audience.

Section icon for: Run log management alongside a SIEMRun log management alongside a SIEM

Organizations with a SIEM installed should consider the benefits and costs of installing a modern log management system to offload the burden of log aggregation and storage from the SIEM. This will enhance the ability to conduct a more comprehensive search with more data sources and longer-retained data.

There are several options for running log management in the same environment as a SIEM. Here are a few to consider.

Collect logs from multiple sources

In this scenario, the log management solution is installed in parallel with the SIEM, and the SIEM continues to use the data it manages for dashboards, reports, and compliance. The log management system is used to store more sources of data to help with investigations, and store it more efficiently to reduce the costs of keeping it for longer.

  1. Keep the data feeds and configuration of the SIEM consistent.

  2. Install the log management system and collect the same data from the endpoints.

  3. Determine if there are data sources that can be offloaded to the log management system to reduce the need for the SIEM to manage or store them.

Collect logs from a data pipeline

Organizations that have deployed a data pipeline have the ability to direct the data to the system that provides the most value for the organization. The benefit of deploying a data pipeline is that it automates the processes involved in extracting, transforming, combining, validating, and loading data for further analysis. It speeds things up by reducing errors and minimizing bottlenecks.

  1. Direct log and event sources into the data pipeline.

  2. Split the data needed for the SIEM and the log management system. In most cases, this is done using Kafka or Nifi workers or filters.

  3. If necessary, use a system like Logstash to move the data into the SIEM and into the log management platform.

  4. Determine if there are data sources that can be offloaded to the log management system to reduce the need for the SIEM to manage or store them.

Collect logs from the SIEM

This scenario may make it quicker to get started using log management. This only works with SIEMs that contain native log forwarding such as QRadar, LogRhythm, McAfee, and RSA. It’s not as efficient as using the log management tool directly, but will offer the ability to begin setting up dashboards and alerts using the same data as the SIEM. It may also be a way for the log management system to store short-term SIEM data for longer periods. For example, a SIEM’s rules may only require 48 hours of data, so there is no need to pay for longer retention in the SIEM if the log management system can store the data more efficiently.

  • Use the SIEM log forwarder to direct the logs to the log management solution. Check configuration in the SIEM settings. This usually requires information like the Hostname or IP address, UDP port, and the format to send the events. Be sure that any routers, firewalls, and security groups allow inbound connections from the SIEM.

  • Use data shippers to transfer data to the log management platforms. Data shippers have many benefits, including retransmitting data on failure, and sending messages in batches. Beats shippers are designed to make sending data easy and reliable.

Use log management to create a data lake

Data lakes are used to store raw enterprise data in one place. The data stored in data lakes aren’t structured or refined, exactly the right kind of place to store log or event files. Data lakes make it easier to work with massive volumes of data coming in at high speed, all in a single repository that serves several use cases. Because the data is left unstructured, it can be stored in inexpensive object storage like Amazon S3.

In this scenario, the log management system collects all the data and stores it efficiently. Data lakes created this way store the data securely, manage access securely, and normalize the data to make it easier to search.

The log management system creates queries that send only the data the SIEM needs.

  1. Collect all data using data shippers and parsers in the log management system.

  2. Create queries that collect the data the SIEM system needs.

  3. Use log forwarding to make the data available to send to the SIEM.

  4. Receive the log files at the SIEM using data shippers like Beats.

Threat Hunting with modern log management

In Use Central Log Management for Security Operations Use Cases, Gartner discuss using Central Log Management (CLM) for threat hunting:

"Threat hunting is also enhanced when there is a CLM capability. Threat hunting is defined as “… an analyst-centric process that enables organizations to uncover hidden advanced threats that were missed by automated preventative and detective controls” (see “Applying Network-Centric Approaches for Threat Detection and Response”). Threat hunting is hard without sufficient data across a wide range of sources and time horizons. To perform quality threat-hunting activities, that data must be collected and stored somewhere. Some considerations when selecting a CLM tool that also has to support threat hunting are:

  • Sufficient storage for the size and type of data that needs to be retained

  • Fast search speed (you don’t want threat hunters waiting hours for a response to a query)

  • Visualization capabilities — not something normally required for basic CLM, but threat hunting is like security operation’s version of business intelligence (BI) and data analysis (see “Predicts 2020: Analytics and Business Intelligence Strategy”)

  • Data enrichment to augment raw data with useful contextual data, such as IP address details (i.e., geolocation, ownership, and registration date)." 1

Hunt Evil Your Practical Guide to Threat Hunting is published by Sqrrl, an AWS company. It provides a complete overview of threat hunting. It offers practical advice for setting up a threat hunting program, and describes steps to follow when practicing threat hunting.2

MITRE ATT&CK is a royalty-free global knowledge base of adversary tactics and techniques based on real-world observations that can help develop specific threat models and methodologies for an organization.3

Section icon for: Get started using Humio with a SIEMGet started using Humio with a SIEM

To understand what’s happening across complex environments, modern log management platforms like Humio provide visibility to streaming logs and event data. By monitoring this type of data the moment it happens, engineers, developers, and security professionals make sure their environment is healthy and performing as expected. When it’s not, they can search through the data to find out exactly what happened and prevent it from causing ongoing problems.

We invite you to see how Humio’s modern architecture redefines what is possible with log management.

Set up a Humio free 30-day trial. See for yourself how Humio can become the foundation for your security system. Our engineers are standing by to help you design a system that helps keep your organization more secure.

Find out more by visiting our website: Humio.com.

To get a free estimate of how much Humio can save your organization, visit our pricing guide.

Unlimited logging with better performance and lower costs

A CUSTOMER STORY

Section icon for: Learn moreLearn more

Website

Humio for enterprise security and SecOps

Why Unlimited

Humio pricing: Modern log management with unmatched Total Cost of Ownership

Humio product information


Product Documentation

Getting started with Humio

Security & Authentication

Corelight Network Security Monitor

Zeek (Bro) Network Security Monitor

Humio data sources

Humio Integrations


Blog posts

Vijilan’s cybersecurity monitoring-as-a-service detects threats in real time with Humio

Streaming live data is the heart of observability

3 ways Humio improves SIEM performance

What is a Data Dead Zone?

Observability and Alerts come together to enable real-time Incident Response and DevOps monitoring

Looking for an Alternative to Splunk, Elasticsearch, Sumo Logic, or Datadog?


Podcasts

Humio at Vijilan with Kevin Nejad, Founder and CEO

Humio with Miguel Adams, Government Agency Security Engineer

Humio Security Developer Kristian Gausel

Corelight with Seth Hall

Security and Bad Code with Jeff Reich of Innove


Customer Case Studies

Be Scord Banking: Fintech Audit Trail Logging with Humio

MSU: Real-Time Observability to Identify and Investigate Security Threats

Netlify: Real-time observability at scale — in all departments

SpareBank 1: Logging everything on-prem and under budget

Humio at Vijilan: Transforming all departments of an MSSP SOC firm and providing new opportunities


Webinars

Real-Time Visibility of Your AWS Systems and Applications

Respond Instantly - Gain Real-Time Insights from Logs, Metrics, Traces, and Events

Building a Modern Observability Stack

Humio Quick Start


Join our Slack channel: meethumio.slack.com

You’ll also find lots of useful information on the Humio blog, and informative talks and demos on the Humio YouTube channel. To hear from Humio developers, customers, and partners, listen to our podcast series: The Hoot.

Section icon for: About HumioAbout Humio

Humio's log management platform offers the lowest total cost of ownership, industry-leading unlimited plans, minimal maintenance and training costs, and remarkably low compute and storage requirements. Humio is the only log management solution that enables customers to log everything to answer anything in real time — at scale, self-hosted or in the cloud. Humio's modern, index-free architecture makes exploring and investigating all data blazing fast, even at scale. Founded in 2016, Humio is headquartered in London and backed by Accel and Dell Technologies Capital.


For more information, visit www.humio.com and follow @MeetHumio on Twitter.

Get alerted to new How To Guides or get a PDF of this one

  1. Gartner: Use Central Log Management for Security Operations Use Cases, Mar 20, 2020, Toby Bussa, Kelly Kavanagh, Mitchell Schneider (Gartner subscription required).

  2. Sqrrl, Hunt Evil Your Practical Guide to Threat Hunting.

  3. MITRE, MITRE ATT&CK.

Start your free trial now, available Self-hosted and SaaS, or request a demo.