Use log management as the foundation of the security stack
Use log management as the foundation of the security stack
With the increase in endpoints, distributed systems, and machine data comes new opportunities for security breaches from inside and outside the organization. Organizations are responsible for protecting their infrastructure, their data, and the private data of their customers. Security teams are responsible for responding when incidents occur, and finding the root-cause to prevent it from happening again.
The toughest security incidents are unforeseen, and often happen where monitoring isn’t in place. How can organizations prepare for the unknown?
With comprehensive and real-time log monitoring of apps, systems, and endpoint data, organizations can minimize recovery time by making it easier to search all relevant data to find the root cause of an incident. This type of visibility gives security teams the continuous insights needed for immediate actions and data-driven responses to strengthen the security across systems, prevent downtime, and protect against attacks.
Modern log management has the ability to collect, analyze, search, and store hundreds of terabytes a day of log, metrics, traces, and events.
Security Information and Event Management (SIEM) platforms are powerful tools for monitoring systems for threats. Companies use SIEMs to protect their company’s data, and to show proof that they are meeting compliance requirements. However, many organizations simply track pre-selected data, leaving blind spots in their monitoring. Due to the ways SIEMs index data, increasing file sizes and processing times, they often make it prohibitively expensive to ingest and store all the data they should.
To solve these challenges, many organizations choose to enhance the capabilities of their SIEM by leveraging log management. This can enhance overall security, save time, and ease budget constraints.
This guide outlines how to center a security stack around log management as a way to increase an organization’s security profile without overloading the limits of the security team or the security solutions already in place. We share the following 5 steps as a way to get started, with more detailed information later in the page.
Integrating log management into security stack
Gartner shares insight into how to think about using log management as part of the security stack in Use Central Log Management for Security Operations Use Cases (Gartner subscription required).
“Central log management is an important, but often undervalued, tool for an organization’s threat management capabilities and compliance requirements. Security and risk management leaders can benefit from the adoption of a log management tool for multiple security operations use cases.
“Modern security operations center activities require access to log data from a variety of sources that may be too expensive to consume in a SIEM solution. However, analysts investigating events may need access to this additional data for context and correlation, and threat hunters need access to a broad scope of data to do their job.” 1
Gartner describes how modern log management is used for security. They provide the following use cases for log management emerging from using Central Log Management (CLM), in addition to what was its primary purpose, IT Operations.
Organizations are using log management to track down root causes and decrease mean time to resolution for security use cases. In our opinion, log management boosts the abilities of a security response team by providing:
Basic threat monitoring with alerts and dashboards
Threat investigations to search correlated events for the root cause of incidents
Threat hunting to search historical data to discover unmonitored threats
Complementing SIEM and MSS/MDR (Managed Security Services/Managed Detection and Response) being used by the organization.
Log management creates a strong foundation for security data
By collecting all appropriate logs and event data, log management provides a strong foundation to support all aspects of data security. With data stored from all endpoints, networks, applications, users, and processes, threats can be identified in seconds, and security incidents can be investigated months into the past.
Modern log management as security platform
Building a security platform using log management provides a cost-efficient way to collect data and events for security alerts, investigations, and compliance. Modern log management technology delivers real-time dashboards and alerts, and offers security teams a streamlined way to search deeper into the past. Organizations large and small use log management to collect logs and events, and use the capabilities of log management to build out the features they need for data security.
A strong foundation to manage data security
Modern log management adds value to an installed SIEM
For organizations with a SIEM platform, modern log management can run alongside a SIEM and provide additional business value. This approach keeps the benefits provided by the existing SIEM tool, such as threat detection, incident response, and reporting. The organization would then use log management to ingest additional data sources. It may be possible to reduce the logs being ingested by the SIEM by collecting them with log management. Because log management platforms are optimized for fast searching and efficient storage, it enhances the organization’s ability to detect, investigate, and manage security data. By reducing the amount of data being managed and retained by the SIEM, this solution can significantly reduce the total cost of managing and storing log and event data.
Together for comprehensive coverage and optimized storage
Challenges managing SIEMs
The implementation of each of these use cases comes with different challenges.
Excessive cost of SIEM log collection
Most SIEM tools are built on data platforms with substantial costs of ingest and retention. This limits the data that’s available for threat detection, alerts, and investigation, and puts SIEMs out of reach for many mid-sized and small enterprise businesses. Log management centralizes all logs and data from across the system, improving exploration and longer retention to meet compliance requirements.
Isolated views of security data
In many organizations, security solutions end up becoming an isolated solution used solely by security operations. As the security team grows, they can drift further from the developer or DevOps team, and lose visibility into the entire attack surface. By using the SIEM together with the central log management platform, both teams can get closer to the data being streamed into both solutions.
Limited data sources
Some SIEM solutions don’t have the ability to collect or manage logs outside of what the solution provides, limiting the scope of the search to what they define is adequate.
Limits on data retention
SIEMs may limit the amount of retention, or charge unreasonable amounts for storage of data beyond 30 or 90 days. Most managed security service providers (MSSPs) store 90 days of data, and may take hours to retrieve it. With a local log management platform, data can be retained at a much lower cost, making it possible to retain data for much longer.
Inflexible license costs
It’s often difficult for SIEM users to find a budgetary sweet spot. They’re often either underinvested in their SIEM and stuck without enough budget to monitor everything in their system, or they’re over-licensed and stuck paying higher costs for services they don’t use. For these SIEM users, modern log management provides a third option: they can scale back their investment in their SIEM and accomplish everything they need to with robust log management at lower license and maintenance costs.
Limited data throughput
SIEMs aren’t optimized for data throughput like modern log management, so they may actively discourage using bandwidth. They may charge based on the number of users, putting pressure on how users interact with the data. Modern log management makes data more freely available around the company, providing additional business value and better insights from data.
Limited security data visibility
A visibility challenge occurs when a suspected incident is detected but the contextual data isn’t available. The security administrator has to go looking outside of the SIEM to understand the scope of the compromise. And often, the data needed is gone because the logs have been rotated. SIEMs on their own provide a limited picture of the system. In order to reduce the load on their already resource-intensive pattern analysis, they use a curated set of log data — so they aren’t seeing the whole picture. Without a robust set of logs and event data, security administrators may become embroiled in what may be a several-week process of manually accessing and reviewing data sources.
Delayed data prevents real-time detection
Many security systems rely on data that is indexed and stored before it’s available for alerts, delaying the time to detection and resolution. Modern log management streams real-time data without indexes, updating alerts in real time, and allowing investigation and searching the moment an incident occurs.
Choose the right log management solution
For the best results, look for a modern log management solution optimized for speed and efficiency. Look for these hallmarks to find the best high-throughput, low-cost system.
5 steps to use log management to monitor and react to security issue
Some companies may choose to use log management without purchasing a SIEM. Modern log management provides a powerful platform for setting up queries and alerts from the same sources used by SIEM tools. In fact, most SIEM tools started as log management tools where data sources, queries, visualizations, and alerts were added over time.
This approach leverages the full power and speed of modern advances in log management platforms, and may pay back the initial time investment. However, it does require security expertise and developer hours to put the system in place. Start with one use case, optimize it, and build the system for additional use cases.
Before building out a security system, it’s important to start by understanding the use cases the system must address. There are a few places to start when deciding which use cases to prioritize.
Ensuring that data is handled correctly and securely is regulated by governments. In many cases, there are specific requirements for creating policies for data governance, keeping data secure, protecting consumer data, and retaining records of compliance for auditing purposes. Log management systems can help collect system, personal, and security logs, and store them to show compliance.
Of course there are countless ways for an external entity to infiltrate an organization and cause harm. Here are a few use cases that should be considered when designing a system to protect data.
Insider threats can be hard to find since it is often conducted by employees with access that seems legitimate. Make sure security use cases take these types of threats into account.
Adversary tactics and techniques and actions to take
For a more detailed list of possible threats integrated with cyber threat intelligence, MITRE publishes a list of tactics used by advisories to infiltrate or harm organizations and steal or destroy their data. They outline tactics, techniques, and information for categories that are part of the cyber kill chain: Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, Collection, Command and Control, Exfiltration, and Impact.
With security use cases prioritized, it’s time to begin collecting data. There are places to start that will become the foundation of security operations.
Threat intelligence is knowledge about threats based on evidence. It contains actionable information on how threats occur, the mechanisms used, context and implications, and specific advice to prevent harm. Threat intelligence can be used to help make decisions about preventing or responding to each threat.
There are several places to get updated threat intelligence, many of which are provided as a community service. Some of these are databases of indicators of compromise, which can be used to alert administrators to possible threats.
Threat intelligence can be used to enhance log data. For example, by looking at new IP addresses and comparing them against a list of bad actors, you can set up an alert when an identified IP address accesses the system.
Open source threat intelligence
There are dozens of open-source options to find information about threats, including these:
AT&T Alien Labs Open Threat Exchange. OTX Pulse provides a summary of threats, a view into the software targeted, and the related indicators of compromise (IOC) that can be used to detect the threats. IOCs include IP addresses, domains hostnames (subdomains), email, URL, URI, file hashes (MD5, SHA1, SHA256, PEHASH, IMPHASH), CIDR rules, file paths, MUTEX name, and CVE number.
Cisco Talos Intelligence IP Reputation Portal has a range of tools designed for the security investigator including IP Reputation.
RiskIQ Community Edition is a portal set up for the community to research security issues using RiskIQ’s extensive data..
Shodan IO scans for a range of Internet devices, breaking them down into industry categories. It is one of the first to focus on IoT devices that are vulnerable or have been violated.
Symantec’s Security Center SPAM Query Tool maintains a list of malware and vulnerabilities, and has an IP check tool for known spammers.
URLhaus – URLhaus is a project operated by abuse.ch. The purpose of the project is to collect, track, and share malware URLs, helping network administrators and security analysts to protect their network and customers from cyber threats.
Zeek is a flexible, open-source network monitoring tool powered by defenders. Zeek interprets what it sees and creates compact, high-fidelity transaction logs, file content, and fully customized output suitable for review with a log management tool or SIEM.
Lists of open source threat intelligence
These sites maintain a list of threat intelligence sources.
Open Source and Other Threat Intelligence Feeds. Barry Raveendran Greene maintains SENKI, a security and resiliency blog. He publishes a list of open-source threat intelligence feeds that are maintained for the participants of the Operator’s Security Toolkit program.
Awesome Threat Intelligence is a list of threat intelligence sources maintained on GitHub by Herman Slatman.
CyberGreen’s Data Source Catalog provided by CyberGreen is a catalog of data sources on cybersecurity risks and vulnerabilities. CyberGreen uses many of these data sources in its stats platform.
To make the data being collected in the log management system more useful for security, set up a dashboard to analyze information about each use case. Use queries to create visualizations that illustrate the results that support the prioritized use case. Create appropriate alerts to notify security personnel when high-profile threat indicators occur that need investigating.
Once a use case has been correctly set up, begin to work on the next use case. As the dashboard is used for analysis, adjust it to make it work for the organization. Optimize the data being collected and the queries driving the visualizations to make achieve the best results for the use case. When the desired results are being successfully achieved for the use case, move to the next prioritized use case.
Continually adjust the dashboard to make it work for the organization. Make sure it illustrates the most important information, and that it helps the business prioritize action. Make sure the data being reported provides value, and that it answers relevant questions for the audience.
Run log management alongside a SIEM
Organizations with a SIEM installed should consider the benefits and costs of installing a modern log management system to offload the burden of log aggregation and storage from the SIEM. This will enhance the ability to conduct a more comprehensive search with more data sources and longer-retained data.
There are several options for running log management in the same environment as a SIEM. Here are a few to consider.
Collect logs from multiple sources
In this scenario, the log management solution is installed in parallel with the SIEM, and the SIEM continues to use the data it manages for dashboards, reports, and compliance. The log management system is used to store more sources of data to help with investigations, and store it more efficiently to reduce the costs of keeping it for longer.
Keep the data feeds and configuration of the SIEM consistent.
Install the log management system and collect the same data from the endpoints.
Determine if there are data sources that can be offloaded to the log management system to reduce the need for the SIEM to manage or store them.
Collect logs from a data pipeline
Organizations that have deployed a data pipeline have the ability to direct the data to the system that provides the most value for the organization. The benefit of deploying a data pipeline is that it automates the processes involved in extracting, transforming, combining, validating, and loading data for further analysis. It speeds things up by reducing errors and minimizing bottlenecks.
Direct log and event sources into the data pipeline.
Split the data needed for the SIEM and the log management system. In most cases, this is done using Kafka or Nifi workers or filters.
If necessary, use a system like Logstash to move the data into the SIEM and into the log management platform.
Determine if there are data sources that can be offloaded to the log management system to reduce the need for the SIEM to manage or store them.
Collect logs from the SIEM
This scenario may make it quicker to get started using log management. This only works with SIEMs that contain native log forwarding such as QRadar, LogRhythm, McAfee, and RSA. It’s not as efficient as using the log management tool directly, but will offer the ability to begin setting up dashboards and alerts using the same data as the SIEM. It may also be a way for the log management system to store short-term SIEM data for longer periods. For example, a SIEM’s rules may only require 48 hours of data, so there is no need to pay for longer retention in the SIEM if the log management system can store the data more efficiently.
Use the SIEM log forwarder to direct the logs to the log management solution. Check configuration in the SIEM settings. This usually requires information like the Hostname or IP address, UDP port, and the format to send the events. Be sure that any routers, firewalls, and security groups allow inbound connections from the SIEM.
Use data shippers to transfer data to the log management platforms. Data shippers have many benefits, including retransmitting data on failure, and sending messages in batches. Beats shippers are designed to make sending data easy and reliable.
Use log management to create a data lake
Data lakes are used to store raw enterprise data in one place. The data stored in data lakes aren’t structured or refined, exactly the right kind of place to store log or event files. Data lakes make it easier to work with massive volumes of data coming in at high speed, all in a single repository that serves several use cases. Because the data is left unstructured, it can be stored in inexpensive object storage like Amazon S3.
In this scenario, the log management system collects all the data and stores it efficiently. Data lakes created this way store the data securely, manage access securely, and normalize the data to make it easier to search.
The log management system creates queries that send only the data the SIEM needs.
Collect all data using data shippers and parsers in the log management system.
Create queries that collect the data the SIEM system needs.
Use log forwarding to make the data available to send to the SIEM.
Receive the log files at the SIEM using data shippers like Beats.
Get started using Humio with a SIEM
To understand what’s happening across complex environments, modern log management platforms like Humio provide visibility to streaming logs and event data. By monitoring this type of data the moment it happens, engineers, developers, and security professionals make sure their environment is healthy and performing as expected. When it’s not, they can search through the data to find out exactly what happened and prevent it from causing ongoing problems.
We invite you to see how Humio’s modern architecture redefines what is possible with log management.
Set up a Humio free 30-day trial. See for yourself how Humio can become the foundation for your security system. Our engineers are standing by to help you design a system that helps keep your organization more secure.
Find out more by visiting our website: Humio.com.
To get a free estimate of how much Humio can save your organization, visit our pricing guide.
Customer Case Studies
You’ll also find lots of useful information on the Humio blog, and informative talks and demos on the Humio YouTube channel. To hear from Humio developers, customers, and partners, listen to our podcast series: The Hoot.
Humio's log management platform offers the lowest total cost of ownership, industry-leading unlimited plans, minimal maintenance and training costs, and remarkably low compute and storage requirements. Humio is the only log management solution that enables customers to log everything to answer anything in real time — at scale, self-hosted or in the cloud. Humio's modern, index-free architecture makes exploring and investigating all data blazing fast, even at scale. Founded in 2016, Humio is headquartered in London and backed by Accel and Dell Technologies Capital.