Use log management as the foundation of the security stack
Use log management as the foundation of the security stack
With the increase in endpoints, distributed systems, and machine data comes new opportunities for security breaches from inside and outside the organization. Organizations are responsible for protecting their infrastructure, their data, and the private data of their customers. Security teams are responsible for responding when incidents occur, and finding the root-cause to prevent it from happening again.
The toughest security incidents are unforeseen, and often happen where monitoring isn’t in place. How can organizations prepare for the unknown?
With comprehensive and real-time log monitoring of apps, systems, and endpoint data, organizations can minimize recovery time by making it easier to search all relevant data to find the root cause of an incident. This type of visibility gives security teams the continuous insights needed for immediate actions and data-driven responses to strengthen the security across systems, prevent downtime, and protect against attacks.
Modern log management has the ability to collect, analyze, search, and store hundreds of terabytes a day of log, metrics, traces, and events.
Security Information and Event Management (SIEM) platforms are powerful tools for monitoring systems for threats. Companies use SIEMs to protect their company’s data, and to show proof that they are meeting compliance requirements. However, many organizations simply track pre-selected data, leaving blind spots in their monitoring. Due to the ways SIEMs index data, increasing file sizes and processing times, they often make it prohibitively expensive to ingest and store all the data they should.
To solve these challenges, many organizations choose to enhance the capabilities of their SIEM by leveraging log management. This can enhance overall security, save time, and ease budget constraints.
This guide outlines how to center a security stack around log management as a way to increase an organization’s security profile without overloading the limits of the security team or the security solutions already in place. We share the following 5 steps as a way to get started, with more detailed information later in the page.
5 Steps to use modern log management for security
Determine security use cases
Prioritize the types of threats that the organization needs to address and prevent, based on the resources available and risk to operations.
Integrating log management into security stack
Gartner shares insight into how to think about using log management as part of the security stack in Use Central Log Management for Security Operations Use Cases (Gartner subscription required).
“Central log management is an important, but often undervalued, tool for an organization’s threat management capabilities and compliance requirements. Security and risk management leaders can benefit from the adoption of a log management tool for multiple security operations use cases.
“Modern security operations center activities require access to log data from a variety of sources that may be too expensive to consume in a SIEM solution. However, analysts investigating events may need access to this additional data for context and correlation, and threat hunters need access to a broad scope of data to do their job.” 1
Gartner describes how modern log management is used for security. They provide the following use cases for log management emerging from using Central Log Management (CLM), in addition to what was its primary purpose, IT Operations.
Organizations are using log management to track down root causes and decrease mean time to resolution for security use cases. In our opinion, log management boosts the abilities of a security response team by providing:
Basic threat monitoring with alerts and dashboards
Threat investigations to search correlated events for the root cause of incidents
Threat hunting to search historical data to discover unmonitored threats
Complementing SIEM and MSS/MDR (Managed Security Services/Managed Detection and Response) being used by the organization.
Log management creates a strong foundation for security data
By collecting all appropriate logs and event data, log management provides a strong foundation to support all aspects of data security. With data stored from all endpoints, networks, applications, users, and processes, threats can be identified in seconds, and security incidents can be investigated months into the past.
Collect and normalize logs and events
Dashboards and alerts
Search across all data
Data retention and long-term storage
Threat dashboards and alerts
Comprehensive threat hunting
Detailed root cause
Dashboards with app and infrastructure status
Longer security data retention
Modern log management as security platform
Building a security platform using log management provides a cost-efficient way to collect data and events for security alerts, investigations, and compliance. Modern log management technology delivers real-time dashboards and alerts, and offers security teams a streamlined way to search deeper into the past. Organizations large and small use log management to collect logs and events, and use the capabilities of log management to build out the features they need for data security.
A strong foundation to manage data security
Modern log management adds value to an installed SIEM
For organizations with a SIEM platform, modern log management can run alongside a SIEM and provide additional business value. This approach keeps the benefits provided by the existing SIEM tool, such as threat detection, incident response, and reporting. The organization would then use log management to ingest additional data sources. It may be possible to reduce the logs being ingested by the SIEM by collecting them with log management. Because log management platforms are optimized for fast searching and efficient storage, it enhances the organization’s ability to detect, investigate, and manage security data. By reducing the amount of data being managed and retained by the SIEM, this solution can significantly reduce the total cost of managing and storing log and event data.
Together for comprehensive coverage and optimized storage
Challenges managing SIEMs
The implementation of each of these use cases comes with different challenges.
Excessive cost of SIEM log collection
Most SIEM tools are built on data platforms with substantial costs of ingest and retention. This limits the data that’s available for threat detection, alerts, and investigation, and puts SIEMs out of reach for many mid-sized and small enterprise businesses. Log management centralizes all logs and data from across the system, improving exploration and longer retention to meet compliance requirements.
Isolated views of security data
In many organizations, security solutions end up becoming an isolated solution used solely by security operations. As the security team grows, they can drift further from the developer or DevOps team, and lose visibility into the entire attack surface. By using the SIEM together with the central log management platform, both teams can get closer to the data being streamed into both solutions.
Limited data sources
Some SIEM solutions don’t have the ability to collect or manage logs outside of what the solution provides, limiting the scope of the search to what they define is adequate.
Limits on data retention
SIEMs may limit the amount of retention, or charge unreasonable amounts for storage of data beyond 30 or 90 days. Most managed security service providers (MSSPs) store 90 days of data, and may take hours to retrieve it. With a local log management platform, data can be retained at a much lower cost, making it possible to retain data for much longer.
Inflexible license costs
It’s often difficult for SIEM users to find a budgetary sweet spot. They’re often either underinvested in their SIEM and stuck without enough budget to monitor everything in their system, or they’re over-licensed and stuck paying higher costs for services they don’t use. For these SIEM users, modern log management provides a third option: they can scale back their investment in their SIEM and accomplish everything they need to with robust log management at lower license and maintenance costs.
Limited data throughput
SIEMs aren’t optimized for data throughput like modern log management, so they may actively discourage using bandwidth. They may charge based on the number of users, putting pressure on how users interact with the data. Modern log management makes data more freely available around the company, providing additional business value and better insights from data.
Limited security data visibility
A visibility challenge occurs when a suspected incident is detected but the contextual data isn’t available. The security administrator has to go looking outside of the SIEM to understand the scope of the compromise. And often, the data needed is gone because the logs have been rotated. SIEMs on their own provide a limited picture of the system. In order to reduce the load on their already resource-intensive pattern analysis, they use a curated set of log data — so they aren’t seeing the whole picture. Without a robust set of logs and event data, security administrators may become embroiled in what may be a several-week process of manually accessing and reviewing data sources.
Delayed data prevents real-time detection
Many security systems rely on data that is indexed and stored before it’s available for alerts, delaying the time to detection and resolution. Modern log management streams real-time data without indexes, updating alerts in real time, and allowing investigation and searching the moment an incident occurs.
Choose the right log management solution
For the best results, look for a modern log management solution optimized for speed and efficiency. Look for these hallmarks to find the best high-throughput, low-cost system.
Architecture for speed, efficiency, and flexibility
Affordable license fees that scale predictably as data requirements grow
Capacity to ingest and store all data required
Streaming data ingest in real time
Fast search with near-zero latency from ingest to being searchable
Easy-to-use free-text search
Data enrichment to augment raw data, including joins from multiple data sources
Dashboards updated in real time
Flexible visualization capabilities
Data compression for efficient storage and data transfer
Long-term retention and storage using inexpensive cloud storage
Resilient design that doesn’t require extensive ongoing maintenance
Self-hosted or SaaS
Self-hosted or SaaS
5 steps to use log management to monitor and react to security issue
Some companies may choose to use log management without purchasing a SIEM. Modern log management provides a powerful platform for setting up queries and alerts from the same sources used by SIEM tools. In fact, most SIEM tools started as log management tools where data sources, queries, visualizations, and alerts were added over time.
This approach leverages the full power and speed of modern advances in log management platforms, and may pay back the initial time investment. However, it does require security expertise and developer hours to put the system in place. Start with one use case, optimize it, and build the system for additional use cases.
Before building out a security system, it’s important to start by understanding the use cases the system must address. There are a few places to start when deciding which use cases to prioritize.
Ensuring that data is handled correctly and securely is regulated by governments. In many cases, there are specific requirements for creating policies for data governance, keeping data secure, protecting consumer data, and retaining records of compliance for auditing purposes. Log management systems can help collect system, personal, and security logs, and store them to show compliance.
Of course there are countless ways for an external entity to infiltrate an organization and cause harm. Here are a few use cases that should be considered when designing a system to protect data.
Insider threats can be hard to find since it is often conducted by employees with access that seems legitimate. Make sure security use cases take these types of threats into account.
Learn more about regulations that have significant impact on data collection and storage.
The General Data Protection Regulation (GDPR) is a European framework that was created to protect security and privacy for Personally Identifiable Information (PII). GDPR applies to any legal entity which stores, controls, or processes personal data for EU citizens.
The Health Insurance Portability and Accountability Act (HIPAA) pertains to organizations that transmit health information in electronic form in the United States. The HIPAA Security Management Process requires organizations to perform risk analysis, risk management, have a policy for data breaches, and conduct Information System Activity Reviews. Logs should be retained for up to six years.
The Payment Card Industry Data Security Standard (PCI DSS) intends to secure credit cardholder data from theft and misuse. There are 12 security areas for enhanced protection for data. It requires collecting system and security logs, and has logging requirements. Specifies the audit trail history retention for at least one year, with a minimum of three months immediately available for analysis.
Learn more about use cases that should be considered to protect data from external threats.
Data exfiltration is when sensitive data is taken outside the organization without authorization. I can be when someone copies data intentionally, or when data is moved as the result of malware.
Connected devices are used by many companies to manage operations, including internet-connected medical equipment, manufacturing machinery and sensors, and more. Internet of Things (IoT) devices often have security gaps.
Detecting and preventing malware is a common security operation. Known malware can be stopped with anti-malware products, but they leave security gaps when new malware is distributed.
Zero-day attacks are software attacks that are unknown and do not have a known fix. These vulnerabilities can begin with a phishing email that contains malware. Once in place, it will often attempt to move laterally to access and exfiltrate sensitive data.
Learn more about the use cases that defend against threats from insiders.
Malicious insiders are employees, former employees, partners, or contractors who use their access to do something illegal or harmful.
Compromised insiders are those who allow someone external to use their credentials.
Privileged access abuse happens when someone has more access to systems than they need to do their jobs, or they have elevated privileges, such as users with Domain Administrator rights or root privileges. Attackers use privileged user credentials to access sensitive information and exfiltrate sensitive data.
Trusted entity compromise happens when an outside attacker takes control of a user’s credentials, often using stealth techniques for months before carrying out an attack.
Adversary tactics and techniques and actions to take
For a more detailed list of possible threats integrated with cyber threat intelligence, MITRE publishes a list of tactics used by advisories to infiltrate or harm organizations and steal or destroy their data. They outline tactics, techniques, and information for categories that are part of the cyber kill chain: Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, Collection, Command and Control, Exfiltration, and Impact.
MITRE ATT&CK can help determine where to start
MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community. ATT&CK is open and available to any person or organization for use at no charge. For more information, see Getting Started with ATT&CK.
The ATT&CK Matrix maps the relationship between threats, tactics, and techniques. It is probably the most widely recognizable aspect of ATT&CK because it is commonly used to show things like defensive coverage of an environment, detection capabilities in security products, and results of an incident or red team engagement.
With security use cases prioritized, it’s time to begin collecting data. There are places to start that will become the foundation of security operations.
Threat intelligence is knowledge about threats based on evidence. It contains actionable information on how threats occur, the mechanisms used, context and implications, and specific advice to prevent harm. Threat intelligence can be used to help make decisions about preventing or responding to each threat.
There are several places to get updated threat intelligence, many of which are provided as a community service. Some of these are databases of indicators of compromise, which can be used to alert administrators to possible threats.
Threat intelligence can be used to enhance log data. For example, by looking at new IP addresses and comparing them against a list of bad actors, you can set up an alert when an identified IP address accesses the system.
Open source threat intelligence
There are dozens of open-source options to find information about threats, including these:
AT&T Alien Labs Open Threat Exchange. OTX Pulse provides a summary of threats, a view into the software targeted, and the related indicators of compromise (IOC) that can be used to detect the threats. IOCs include IP addresses, domains hostnames (subdomains), email, URL, URI, file hashes (MD5, SHA1, SHA256, PEHASH, IMPHASH), CIDR rules, file paths, MUTEX name, and CVE number.
Cisco Talos Intelligence IP Reputation Portal has a range of tools designed for the security investigator including IP Reputation.
RiskIQ Community Edition is a portal set up for the community to research security issues using RiskIQ’s extensive data..
Shodan IO scans for a range of Internet devices, breaking them down into industry categories. It is one of the first to focus on IoT devices that are vulnerable or have been violated.
Symantec’s Security Center SPAM Query Tool maintains a list of malware and vulnerabilities, and has an IP check tool for known spammers.
URLhaus – URLhaus is a project operated by abuse.ch. The purpose of the project is to collect, track, and share malware URLs, helping network administrators and security analysts to protect their network and customers from cyber threats.
Zeek is a flexible, open-source network monitoring tool powered by defenders. Zeek interprets what it sees and creates compact, high-fidelity transaction logs, file content, and fully customized output suitable for review with a log management tool or SIEM.
Lists of open source threat intelligence
These sites maintain a list of threat intelligence sources.
Open Source and Other Threat Intelligence Feeds. Barry Raveendran Greene maintains SENKI, a security and resiliency blog. He publishes a list of open-source threat intelligence feeds that are maintained for the participants of the Operator’s Security Toolkit program.
Awesome Threat Intelligence is a list of threat intelligence sources maintained on GitHub by Herman Slatman.
CyberGreen’s Data Source Catalog provided by CyberGreen is a catalog of data sources on cybersecurity risks and vulnerabilities. CyberGreen uses many of these data sources in its stats platform.
To make the data being collected in the log management system more useful for security, set up a dashboard to analyze information about each use case. Use queries to create visualizations that illustrate the results that support the prioritized use case. Create appropriate alerts to notify security personnel when high-profile threat indicators occur that need investigating.
Once a use case has been correctly set up, begin to work on the next use case. As the dashboard is used for analysis, adjust it to make it work for the organization. Optimize the data being collected and the queries driving the visualizations to make achieve the best results for the use case. When the desired results are being successfully achieved for the use case, move to the next prioritized use case.
Continually adjust the dashboard to make it work for the organization. Make sure it illustrates the most important information, and that it helps the business prioritize action. Make sure the data being reported provides value, and that it answers relevant questions for the audience.
Run log management alongside a SIEM
Organizations with a SIEM installed should consider the benefits and costs of installing a modern log management system to offload the burden of log aggregation and storage from the SIEM. This will enhance the ability to conduct a more comprehensive search with more data sources and longer-retained data.
There are several options for running log management in the same environment as a SIEM. Here are a few to consider.
Collect logs from multiple sources
In this scenario, the log management solution is installed in parallel with the SIEM, and the SIEM continues to use the data it manages for dashboards, reports, and compliance. The log management system is used to store more sources of data to help with investigations, and store it more efficiently to reduce the costs of keeping it for longer.
Keep the data feeds and configuration of the SIEM consistent.
Install the log management system and collect the same data from the endpoints.
Determine if there are data sources that can be offloaded to the log management system to reduce the need for the SIEM to manage or store them.
Collect logs from a data pipeline
Organizations that have deployed a data pipeline have the ability to direct the data to the system that provides the most value for the organization. The benefit of deploying a data pipeline is that it automates the processes involved in extracting, transforming, combining, validating, and loading data for further analysis. It speeds things up by reducing errors and minimizing bottlenecks.
Direct log and event sources into the data pipeline.
Split the data needed for the SIEM and the log management system. In most cases, this is done using Kafka or Nifi workers or filters.
If necessary, use a system like Logstash to move the data into the SIEM and into the log management platform.
Determine if there are data sources that can be offloaded to the log management system to reduce the need for the SIEM to manage or store them.
Collect logs from the SIEM
This scenario may make it quicker to get started using log management. This only works with SIEMs that contain native log forwarding such as QRadar, LogRhythm, McAfee, and RSA. It’s not as efficient as using the log management tool directly, but will offer the ability to begin setting up dashboards and alerts using the same data as the SIEM. It may also be a way for the log management system to store short-term SIEM data for longer periods. For example, a SIEM’s rules may only require 48 hours of data, so there is no need to pay for longer retention in the SIEM if the log management system can store the data more efficiently.
Use the SIEM log forwarder to direct the logs to the log management solution. Check configuration in the SIEM settings. This usually requires information like the Hostname or IP address, UDP port, and the format to send the events. Be sure that any routers, firewalls, and security groups allow inbound connections from the SIEM.
Use data shippers to transfer data to the log management platforms. Data shippers have many benefits, including retransmitting data on failure, and sending messages in batches. Beats shippers are designed to make sending data easy and reliable.
Use log management to create a data lake
Data lakes are used to store raw enterprise data in one place. The data stored in data lakes aren’t structured or refined, exactly the right kind of place to store log or event files. Data lakes make it easier to work with massive volumes of data coming in at high speed, all in a single repository that serves several use cases. Because the data is left unstructured, it can be stored in inexpensive object storage like Amazon S3.
In this scenario, the log management system collects all the data and stores it efficiently. Data lakes created this way store the data securely, manage access securely, and normalize the data to make it easier to search.
The log management system creates queries that send only the data the SIEM needs.
Collect all data using data shippers and parsers in the log management system.
Create queries that collect the data the SIEM system needs.
Use log forwarding to make the data available to send to the SIEM.
Receive the log files at the SIEM using data shippers like Beats.
Threat Hunting with modern log management
In Use Central Log Management for Security Operations Use Cases, Gartner discuss using Central Log Management (CLM) for threat hunting:
"Threat hunting is also enhanced when there is a CLM capability. Threat hunting is defined as “… an analyst-centric process that enables organizations to uncover hidden advanced threats that were missed by automated preventative and detective controls” (see “Applying Network-Centric Approaches for Threat Detection and Response”). Threat hunting is hard without sufficient data across a wide range of sources and time horizons. To perform quality threat-hunting activities, that data must be collected and stored somewhere. Some considerations when selecting a CLM tool that also has to support threat hunting are:
Sufficient storage for the size and type of data that needs to be retained
Fast search speed (you don’t want threat hunters waiting hours for a response to a query)
Visualization capabilities — not something normally required for basic CLM, but threat hunting is like security operation’s version of business intelligence (BI) and data analysis (see “Predicts 2020: Analytics and Business Intelligence Strategy”)
Data enrichment to augment raw data with useful contextual data, such as IP address details (i.e., geolocation, ownership, and registration date)." 1
Hunt Evil Your Practical Guide to Threat Hunting is published by Sqrrl, an AWS company. It provides a complete overview of threat hunting. It offers practical advice for setting up a threat hunting program, and describes steps to follow when practicing threat hunting.2
MITRE ATT&CK is a royalty-free global knowledge base of adversary tactics and techniques based on real-world observations that can help develop specific threat models and methodologies for an organization.3
Determine security use cases and access the data sources they require.
Determine what insights the organization requires, the data that is needed, and how that information will be provided with analytics, visualizations, and alerts. Prioritize data sources based on what’s mandatory (for example those needed for compliance or intrusion detection) and what will provide additional business value.
Determine storage requirements for data.
Determine what data is needed, and the time needed for active dashboards and alerts. Breaches can occur months after a system has been compromised, so determine how long data should be collected to assist in investigations. Identify the required retention time of data stored for compliance purposes — some data stored for compliance must be available for at least a year. A sophisticated log management platform that optimizes storage with compression may be able to store 10-20x or more than the SIEM. Cloud storage may offer a cost that makes storing data for years an affordable option.
Deploy log management in the same environment as the logs.
If logs are being generated from platforms, services, or apps in the cloud, it may be required to pay to send logs to or from the cloud. Deploy a log management system in the same cloud platform to minimize data transfer costs.
Account for hidden costs of open-source tools.
While SIEMs based on open-source platforms may look attractive at the onset, they come with additional requirements to install, manage, and customize that may end up being much more expensive than what was saved on the license.
Optimize for threat hunting.
Using log management as the centerpiece of the security stack makes it easier to become a skilled threat hunter. There may be logs and other event data that is pertinent to threat hunting that isn’t collected or saved in a SIEM system. Modern log management platforms that allow unlimited logging and long retention are ideal because they provide centralized, detailed logs that can be searched.
Leverage existing log management tools from other organizations for SecOps.
Before spending more to increase data volumes or add features for an existing SIEM, see if there are organizations already collecting logs from the IT environment. They may be open to supporting security use cases with data they may already be collecting.
Get started using Humio with a SIEM
To understand what’s happening across complex environments, modern log management platforms like Humio provide visibility to streaming logs and event data. By monitoring this type of data the moment it happens, engineers, developers, and security professionals make sure their environment is healthy and performing as expected. When it’s not, they can search through the data to find out exactly what happened and prevent it from causing ongoing problems.
We invite you to see how Humio’s modern architecture redefines what is possible with log management.
Set up a Humio free 30-day trial. See for yourself how Humio can become the foundation for your security system. Our engineers are standing by to help you design a system that helps keep your organization more secure.
Find out more by visiting our website: Humio.com.
To get a free estimate of how much Humio can save your organization, visit our pricing guide.
Unlimited logging with better performance and lower costs
A CUSTOMER STORY
Customer Case Studies
You’ll also find lots of useful information on the Humio blog, and informative talks and demos on the Humio YouTube channel. To hear from Humio developers, customers, and partners, listen to our podcast series: The Hoot.
Humio's log management platform offers the lowest total cost of ownership, industry-leading unlimited plans, minimal maintenance and training costs, and remarkably low compute and storage requirements. Humio is the only log management solution that enables customers to log everything to answer anything in real time — at scale, self-hosted or in the cloud. Humio's modern, index-free architecture makes exploring and investigating all data blazing fast, even at scale. Founded in 2016, Humio is headquartered in London and backed by Accel and Dell Technologies Capital.
Get alerted to new How To Guides or get a PDF of this one
Gartner: Use Central Log Management for Security Operations Use Cases, Mar 20, 2020, Toby Bussa, Kelly Kavanagh, Mitchell Schneider (Gartner subscription required).
MITRE, MITRE ATT&CK.