[object Object]

Routing Instana Events to Humio

Correlating APM performance data from Instana with log data from Humio helps teams build better software faster.

Logging is a natural part of coding. Believe it or not, we've all programmed logging into our code. After all, who hasn't written a "Hello World" program?

Hello world

It's not just your code that writes out log messages, every other piece of software does, from the operating system all the way up the stack. Tools to ship, aggregate and index logs have been around for a long time; it's a mature market. That does not mean that there is no room for a new startup to disrupt the market. Humio received $9M in Series A funding from Accel at the beginning of 2019 and is aiming to do just that. The Humio log management platform is lightning fast, flexible, and built to scale - all at an affordable price. Integrating data sources between Humio and Instana is useful because DevOps, IT Ops and Security professionals need many types of data and information to optimise their applications and speed up software development. Correlating APM performance data with log data helps teams build better software faster.

Humio has many standard data ingesters, making it easy to plumb into your existing configuration. For viewing the data, there are dashboards and an API for querying and exporting data.

Choose channel type

Instana already has several built-in integrations for Alert Channels to send notifications for distribution to various other systems. It’s easy to push Instana events into Humio via a log routing tool such as Logstash or Fluentd then forward them to Humio. Alternatively, directly via the Splunk compatible HTTP Events Collector (HEC).

Once you have Instana events streaming into Humio, you can correlate across multiple event stream sources. For example, using Jenkins as your CI/CD delivery pipeline, automating your builds and deployments into Kubernetes. It is possible to correlate deployment events from Jenkins with service quality events from Instana to verify that new deployments do not have a negative performance impact.

Each Instana event includes a contextual deep link back to the Instana dashboard, enabling you to start root cause analysis immediately.

Fire it up

If you’re already using a log router shipper in your environment, it’s easy to plumb in Instana events. Alternatively, go direct via the HEC.

Logstash and Fluentd can be installed natively or run inside a container. For this article, I used containers running inside a Kubernetes (GKE) cluster. I have also assumed that you have your Humio instance up and running, either on SaaS, as a container or natively installed.

Fluentd

For the Fluentd container, you will need to take the base image and add a few extra bits. Here is the Dockerfile I used.

FROM fluent/fluentd:v1.4.2-debian-2.0
USER root
RUN apt-get update && \
    apt-get install -y build-essential ruby-dev
RUN fluent-gem install fluent-plugin-elasticsearch && \
    fluent-gem install fluent-plugin-elasticsearch-timestamp-check

Once the new Docker image is built and pushed to a repository it's easy to spin it up inside your Kubernetes cluster. These are the deployment files I used, you’ll need to edit them and substitute your values. Examining the configuration map for the fluentd.conf file you can see the input configuration for the Webhook endpoint and the output configuration to push the events into Humio via the Elasticsearch bulk API.

That's all there is to it.

Logstash

For the Logstash container, we can use the one from Docker Hub without modification.

These are the deployment files I used, you’ll need to edit them and substitute your values. Examining the configuration map for the logstash-config file, you can see that it is very similar to the Fluentd file, just with a different syntax.

Instana

To have Instana send events to your configured log router. Configure an Alert Channel WebHook to send events to your Fluentd/Logstash HTTP endpoint. Next configure Alerting to send the selected alerts through the previously configured Alert Channel. For testing, set Events to “Alert on Event Type(s)”, select all Types and set Scope to “All Available Entities”. As long as there is activity in the application environment you are monitoring with Instana, those events will be propagated to the configured logging aggregator.

Direct

Humio has Splunk compatible HEC end point. Just configure Instana’s built in Splunk Alert Channel to point to your Humio instance.

Events on Instana dashboard Events on Instana dashboard

Humio showing Instana events Humio showing Instana events

The Best of Both Worlds

By utilising a log router such as Fluentd or Logstash, or going direct it is simple to get Instana events into Humio. Because each event contains a deep link back into Instana, drilling down from the event is simple, just a case of following the link.

Instana has deep links not just for events, a Dynamic Focus query can be passed as a query parameter.

http://prod-acme.instana.io/#/physical?q=entity.docker.containerId:a1b2c3d4e5f6

http://prod-acme.instana.io/#/physical?q=entity.host.name:foo

Using deep links like these provides other drill downs from Humio or Grafana dashboards into Instana.

To make the link look nice on a Humio dashboard use:

format("[%s](%s)", field=[@issue.text, @issue.link], as=link) | table([link, @issue.type, @issue.suggestion])

Note the use of Markdown link syntax .

Take advantage of what Humio has to offer and get started today with our free trial, or schedule a live demo with a Humio team member.