Routing Instana Events to Humio
Correlating APM performance data from Instana with log data from Humio helps teams build better software faster.
Logging is a natural part of coding. Believe it or not, we've all programmed logging into our code. After all, who hasn't written a "Hello World" program?
It's not just your code that writes out log messages, every other piece of software does, from the operating system all the way up the stack. Tools to ship, aggregate and index logs have been around for a long time; it's a mature market. That does not mean that there is no room for a new startup to disrupt the market. Humio received $9M in Series A funding from Accel at the beginning of 2019 and is aiming to do just that. The Humio log management platform is lightning fast, flexible, and built to scale - all at an affordable price. Integrating data sources between Humio and Instana is useful because DevOps, IT Ops and Security professionals need many types of data and information to optimise their applications and speed up software development. Correlating APM performance data with log data helps teams build better software faster.
Humio has many standard data ingesters, making it easy to plumb into your existing configuration. For viewing the data, there are dashboards and an API for querying and exporting data.
Instana already has several built-in integrations for Alert Channels to send notifications for distribution to various other systems. It’s easy to push Instana events into Humio via a log routing tool such as Logstash or Fluentd then forward them to Humio. Alternatively, directly via the Splunk compatible HTTP Events Collector (HEC).
Once you have Instana events streaming into Humio, you can correlate across multiple event stream sources. For example, using Jenkins as your CI/CD delivery pipeline, automating your builds and deployments into Kubernetes. It is possible to correlate deployment events from Jenkins with service quality events from Instana to verify that new deployments do not have a negative performance impact.
Each Instana event includes a contextual deep link back to the Instana dashboard, enabling you to start root cause analysis immediately.
Fire it up
If you’re already using a log router shipper in your environment, it’s easy to plumb in Instana events. Alternatively, go direct via the HEC.
Logstash and Fluentd can be installed natively or run inside a container. For this article, I used containers running inside a Kubernetes (GKE) cluster. I have also assumed that you have your Humio instance up and running, either on SaaS, as a container or natively installed.
For the Fluentd container, you will need to take the base image and add a few extra bits. Here is the Dockerfile I used.
FROM fluent/fluentd:v1.4.2-debian-2.0 USER root RUN apt-get update && \ apt-get install -y build-essential ruby-dev RUN fluent-gem install fluent-plugin-elasticsearch && \ fluent-gem install fluent-plugin-elasticsearch-timestamp-check
Once the new Docker image is built and pushed to a repository it's easy to spin it up inside your Kubernetes cluster. These are the deployment files I used, you’ll need to edit them and substitute your values. Examining the configuration map for the fluentd.conf file you can see the input configuration for the Webhook endpoint and the output configuration to push the events into Humio via the Elasticsearch bulk API.
That's all there is to it.
For the Logstash container, we can use the one from Docker Hub without modification.
These are the deployment files I used, you’ll need to edit them and substitute your values. Examining the configuration map for the logstash-config file, you can see that it is very similar to the Fluentd file, just with a different syntax.
To have Instana send events to your configured log router. Configure an Alert Channel WebHook to send events to your Fluentd/Logstash HTTP endpoint. Next configure Alerting to send the selected alerts through the previously configured Alert Channel. For testing, set Events to “Alert on Event Type(s)”, select all Types and set Scope to “All Available Entities”. As long as there is activity in the application environment you are monitoring with Instana, those events will be propagated to the configured logging aggregator.
Events on Instana dashboard
Humio showing Instana events
The Best of Both Worlds
By utilising a log router such as Fluentd or Logstash, or going direct it is simple to get Instana events into Humio. Because each event contains a deep link back into Instana, drilling down from the event is simple, just a case of following the link.
Instana has deep links not just for events, a Dynamic Focus query can be passed as a query parameter.
Using deep links like these provides other drill downs from Humio or Grafana dashboards into Instana.
To make the link look nice on a Humio dashboard use:
format("[%s](%s)", field=[@issue.text, @issue.link], as=link) | table([link, @issue.type, @issue.suggestion])