Why Humio created a modern purpose-built logging tool from scratch
Kresten Krab Thorup shares the story behind the creation of Humio
October 16th, 2019
In a recent Hoot podcast episode Time Series Databases with Kresten Krab Thorup, PJ Haggerty sat down with Humio founder and CTO Kresten Krab Thorup to discuss his experience building the Humio log management platform.
High-ingesting, high-performance logging platforms like Humio don’t spring into existence overnight. Kresten recalls that the ideas that led to the formation of Humio grew out of a series of insights.
Prior to founding Humio, Kresten and his team tried several different platforms to meet their logging needs, including ELK and Splunk. They were frustrated with the price, operation, and functionally of what was available. Specifically, they wanted to access and trigger events from logs in real time. After surveying the field and not finding what they needed, they concluded, “We can do better.”
Start with an index-free timestamp database
Kresten was startled by the inefficiencies of traditional database systems. Using a traditional database, he’d burn significant processing power indexing incoming data, and then use up even more memory accessing data on disk. They quickly learned that for typical incident response queries, only 2% of data taken in was ever looked again, so they focused on making the queries run faster, rather than wasting resources on building indexes.
So Kresten and his team started Haystack — the project that would evolve into Humio — with a focus on the efficiencies of timestamp databases over traditional databases. They began by building a database using a timestamp followed by a series of schematized numbers. Eventually, they found that a timestamp followed by text could be more efficient, so they transitioned to that kind of system and named it Humio.
Inspired by Mechanical Sympathy
As they built these new platforms, Kresten took inspiration from the real-time computation work of high-frequency trading on the stock market. Fascinated by the efficiencies built into those systems, his team of engineers turned their attention to maximizing their CPU usage by keeping their data pipeline full and making the most of in-memory processing techniques. “We were heavily influenced by work on mechanical sympathy, that you can write programs that really leverage modern hardware,” Kresten recalls.
Humio is purpose-built for real-time incident response and debugging
To tailor Humio for efficiency, Kresten and his team identified its main use cases: reacting to something happening in real time, and incident response or debugging. Kresten observes how these low-query, time-specific scenarios save processing power, “The insight was that when you do incident response, typically you know when it happened. With that starting point, that reduces the data you need to sift through.”
As Humio developed, some companies were impressed with Humio’s performance and began requesting features that didn’t leverage the architecture of timestamp databases. “In that world, most data is heavily schematized. Humio is much more open-ended text processing,” explains Kresten. Instead of diluting the quality of their platform, the team turned down these cases, doubling down on optimizing Humio for high-volume time-series databases. In doing so, they made Humio into a powerful tool in a world where timestamp-data-reporting smart devices are proliferating, and time-series databases are the fastest-growing type of database in popularity.
Humio modern architecture unlocks unlimited ingest at blazing speeds
Because of years of active development by some of the smartest minds in the industry, Humio is known for being purpose-built for the scale of today’s data volumes. While traditional solutions continue to limit access to customer’s data through pre-determined views, or limits sets to just samples of data, Humio changes how organizations log by enabling users to log everything to answer anything, in real time. Humio unlocks unlimited ingest without complexity, and at a fraction of the price of other solutions.
By removing obstacles to give customers autonomy, Humio enables its customers to determine their logging practices without concerns for restrictive technical, hardware or financial resources.
Learn more about Humio, and get started with a demo or free trial
Visit Humio.com to find out how Humio’s purpose-built log management platform enables live system observability through fast, scalable, and efficient log data management. Sign up for an online demo or download a free 30-day trial to see what Humio can do for you.