Chaos Observability with Humio
This post was originally posted on Medium on February 21.
Building on yesterday’s article that showed how the Chaos Toolkit could now surface the commands and progress of chaos experiments into Slack, amongst other places, using the new notifications support, today it was time to take this support and integrate it into the cloud native observability solution, Humio.
Introducting the new incubator project: ChaosToolkit-Humio
When you add chaostoolkit-humio to your Chaos Toolkit python environment, the extension automatically listens to all the Chaos Toolkit notifications and sends them as events to Humio:
Now you can inspect current and past events in Humio and answer questions such as “What chaos was running on what systems at that point in time?”.
Through the new Humio extension and integration, Chaos Toolkit chaos experiments can now be brought into your overall log aggregation and observability strategy.
Next, we plan to take the Humio integration further so that we can introduce probes for you to inspect, as part of your actual chaos experiments, the output that appears in Humio. When this is done, you’ll be able to incorporate those observations into your chaos experiments taking full advantage of the huge quantities of logs available in a typical Humio deployment, sourced from all over your production system.
Thanks to Martin W. Lassen.