Improve application monitoring with modern log management

Contents

Microservices have changed everything. To be more specific, the challenges of managing microservices has changed everything. Whether talking about massive application complexity, the dynamic aspect of services, or the melding of infrastructure and software platforms into wholly contained tech stacks, the days of using a single tool to monitor application performance are over.

At the same time, the IT stack is becoming more intertwined with the entire business operation. Non-IT individuals now use application information to answer critical business questions specific to their teams, like seeing how the network or databases are supporting a widely-used app, knowing the status of the app to answer customer support questions, or knowing what features are being used to better support marketing efforts.

This “Brave New World” of applications means that to understand the entire value proposition of applications requires the best of both worlds — Application Performance Monitoring (APM) tools and log management solutions working together to answer questions for every stakeholder.

With microservices and cloud applications comes new challenges

Understanding the impact of microservice architectures and the technology stack on IT Operations teams requires a little understanding of the application technology timeline — and why microservices represents both an expected and unexpected shift in the stack.

As Java rose to prominence in the enterprise application world, the lack of production visibility became a major issue, especially as banking, insurance, and telco providers began rolling out massively-scaled (at the time) applications for their customers.

Thus entered the first generation of APM tools, built on the ability to inject bytecode into the running applications to monitor component-level performance.

Over time, a new breed of enterprise applications emerged — Service Oriented Architecture (SOA) applications. This paved the path for a second generation of APM solutions that could deal with the distributed nature of SOA applications, adding “maps” to the vocabulary of IT operations. But even as an entire new generation of tools emerged, the primary ideal of technology was inserting bytecode for monitoring into applications.

The growth of microservice applications over the last four years has created an unprecedented explosion in IT organizations when it comes to how applications are built, deployed, and operated.

  • Large percentages of microservices that make up applications aren’t running code to instrument.

  • Each developer (and development team) has the capacity to make their own platform and infrastructure decisions on everything from language choice to selecting the specific database and messaging services to include.

  • Dynamism — constant change in application paths — continues to grow, making it more difficult to capture and understand the dependencies within an application environment.

A third generation of APM tools have surfaced to help monitor these new, highly-distributed, and dynamic applications. But the three changes discussed above also make it more difficult for APM users to deliver the back-half of APM value— solving problems when they occur.

This guide outlines how Log Management solutions and APM tools are perfect complements for each other. We share the following 5 steps as a way to get started, with more detailed information later in the page.

5 How To's

5 Steps to use log management to strengthen APM

Challenges in managing microservice apps

Why is it so difficult to manage microservice applications? While the nature of microservice technology creates difficulties in managing applications, there’s also an organizational shift that must be dealt with to meet the ultimate needs of the business with their applications — the need to move fast.

Continuous change

About the only thing that’s constant in today’s application operations is that there is continuous change. High-performing organizations no longer measure their application operations by the number of releases per year — instead, they focus on the number of updates per day. This continuous delivery model delivers higher quality and higher performance overall, but it creates all kinds of problems for monitoring, especially since the biggest challenge of application management is the need to understand context.

Complexity

Of course, regardless of how dynamic an environment is, who gets involved in application operations, or how often software is updated, application complexity is the biggest roadblock to all stakeholders getting the most out of their apps.

  1. How can individuals (or even teams) possibly see the operations, relationships, and dependencies of all the entities running in an application environment?

  2. How can they begin to understand and interpret how millions of pieces of information indicate whether an application is running well, and how users are being served?

More stakeholders

The days of an app server admin and a chief architect being the only two people involved with application monitoring are long gone. The need for speed encompasses the entire organization – developers, architects, operations, QA – even application business owners. After all, if you’re creating the ability to quickly adjust to market forces in your application, then those team members tasked with monitoring the market must be stakeholders in overall application success (which is more than just performance).

Limited data

APM solutions rely on their monitoring agents to get the data into the system for analysis and reporting. But there are plenty of pieces of data – both configuration and operational – that exist in logs but can’t be brought into the system via a monitoring agent. Logging management solutions can access more data from specific platforms than APM monitoring agents can get, including network issues, database connections or availability, or information about what’s happening in a container that the app relies on.

Unsynced timestamps

It can be difficult to research data from multiple sources via timestamps. It’s not that APM tools can’t sync on time; rather, this is difficult if additional data isn’t laid out in the same time frame as deep application monitoring — the information that provides the critical details for solving issues, especially those associated with configuration or platform dependencies.

Siloed monitoring solutions

The same categories of data from APM (time sequences, configuration information, updates, performance issues, resource usage, etc.) are also the hallmarks of other monitoring tools (network performance monitoring, server monitoring, deep user monitoring, etc.). Often, one affects the other, and the cause of an issue under investigation may not be collected by the APM. The best-situated platform for looking at all monitoring data in aggregate is the log management solution. By looking at data from the APM or log management alone, it may take longer to discover the cause of a performance issue. By bringing other operational details to bear, log management can take things to the next level.

But wait! There’s more!

Using an APM in isolation leaves out other operational tools that produce useful logs – from PagerDuty to Jenkins. All of them have timestamps, warnings, alerts, and other messages. They help to understand just how the application is running, and how it is operating in other parts of the infrastructure. The only common denominators between this completely-unconnected set of tools are logging and timestamps.

Log management and APM: better together

Log Management solutions and APM tools are perfect complements for each other — operating on adjacent technology layers. APM solutions optimize analysis of specialized data to answer a discrete set of questions about applications, while log management tools use less specialized but more comprehensive data, and a user interface designed for a broader set of questions.

A bonus of this complementary pair is the fact that log management tools can ingest data from APM solutions, making APM data available for broader analysis capabilities.

Choose the right log management solution

The nature of applications and application troubleshooting means that not every log management solution can be brought to bear on microservice applications. There are at least three absolute requirements for a log management solution to be truly helpful to a complex APM tool, especially when dealing with microservices.

  1. Unlimited data ingestion
    With microservices there is exponentially more data than monolithic or SOA applications. On top of the individual stack data, there’s also data available from the applications, and each request can have a unique path through the infrastructure. Trying to guess what pieces of data to include for analysis isn’t just a difficult proposition, it’s practically impossible. This is why microservice application monitoring solutions put so much emphasis on automation and mapping — because there’s too much for an individual (or team) to take in and understand.

  2. Non-indexed queries.
    In a world where time is the arbiter of success, the need to index data on the way in and query indexes for analysis simply gets in the way of advanced data analysis. And as situations change, the next query will have to build on the last. Just one troubleshooting session could incorporate dozens of queries. If streaming data can be collected without being restricted to defining the schema up front, there is much more freedom to explore relationships later. And when a search is easily generated and results come in instantly, it encourages the user to ask more questions and explore further.

  3. Real-time data and streaming.
    Yes, this is technically two different items, but they’re related enough to think of them together. As organizations move from a few software releases a year to dozens every day, the need for immediate feedback is greater than ever. The only way to effectively assist the ops team to keep their service levels up is to provide data in near real time. The best way to do that is to stream data from the source and process it without delaying for indexing.

For the best results, look for a modern log management solution optimized for speed and efficiency. Look for these hallmarks to find the best high-throughput, low-cost log management system.

5 steps to troubleshoot apps with log management and APM together

One critical aspect of microservices and container-based applications is the percentage of service platforms that aren’t running custom code. Instead, they provide a critical function to the overall operation — including database, security, storage, and messaging, to name a few.

Troubleshooting and problem-solving in this environment requires the ability to watch the interactions between systems, as opposed to simply stepping through custom code. These are the steps for isolating the root cause of microservice application problems in these distributed environments.

Tips & Tricks

Think AEM: Application “Effectiveness” Monitoring.

It’s not about monitoring application performance – it’s about managing application effectiveness. A broader set of stakeholders actually need something different. They’re more interested in how applications are delivering their promised (or desired) value – to customers, to partners, to sales teams, and to the business. That’s “application effectiveness.” That’s the value proposition that you get from putting the right application architecture, APM solution, and Analytics solution together for your entire team.

Correlate across multiple event streams.

Once you have the APM events streaming into log management, you can correlate across multiple event stream sources. For example, if you use Jenkins as your CI/CD delivery pipeline to automate builds and deployments into Kubernetes, it is possible to correlate deployment events from Jenkins with service quality events from the APM to verify that new deployments do not have a negative performance impact. In some cases, the APM event includes a contextual deep link back to the APM dashboard, enabling you to start root cause analysis immediately.

Supercharge APM trouble tickets.

If you’re using trouble tickets to assign APM use to log management, make sure that the IP address or server name of the APM-identified system is in the ticket, so that anyone can pick up the work at any time in the future.

Think like a hub.

Airlines run a spoke and hub flight operations system – it allows them to be more efficient by running all passengers through a small collection of hubs. In a complex application system, every service (machine, server, network device, etc.) is its own hub, whether architected to an official hub or not. There are a set of requests coming into the service for work to be done – upstream requests. There are requests made by the service for work it needs by other systems – downstream requests.

Isolating the true root cause of a problem requires the ability to analyze an application component from both perspectives. Where are requests coming from – and what resources are they consuming? What requests are being made by the component to other services or components – and what is the status and latency of those requests?

Complete the circle.

Don’t stop when the root cause is found, complete the circle. Once you’ve isolated the issue, flip the focus of the search. Go back up the stack to see what else that issue might be affecting. Is the same issue causing problems with other apps? Is there something they need to do to mitigate any problems?

Start a journal.

In case you can’t solve the problem on the first occurrence, log the time and situation for each time it occurs. This brings you one step closer to finding and fixing the problem. The journal will help you as you search for the pattern around the issues.

Getting started using an APM with Humio

The Humio log management platform is lightning fast, flexible, and built to scale – all at an affordable price. Integrating data sources between Humio and Instana is useful because DevOps, IT Ops, and Security professionals need many types of data and information to optimize their applications and speed up software development. Correlating APM performance data with log data helps teams build better software faster.

The following steps show how to configure Instana APM to work with Humio.

We invite you to see how Humio’s modern architecture redefines what is possible with log management. Request a free live demo to find out how Humio can help your organization improve the quality of app development and reduce infrastructure monitoring costs with modern log management.

Set up a Humio free 30-day trial. See for yourself how Humio can enhance the value of application performance monitoring.

About Humio

Humio's log management platform offers the lowest total cost of ownership, industry-leading unlimited plans, minimal maintenance and training costs, and remarkably low compute and storage requirements. Humio is the only log management solution that enables customers to log everything to answer anything in real time — at scale, self-hosted or in the cloud. Humio's modern, index-free architecture makes exploring and investigating all data blazing fast, even at scale. Founded in 2016, Humio is headquartered in London and backed by Accel and Dell Technologies Capital.

For more information, visit www.humio.com and follow @MeetHumio on Twitter.