Live Workshop icon

April 30: Setting up a modern observability stack with Garland Technology, Corelight, and Humio

Humio now features joins, query quotas, a new chart engine, and UI updates

New capabilities and improvements make Humio even more powerful

February 20th, 2020

Humio is excited to announce the general availability of several new features in the Humio platform, for on-premise and Humio Cloud users. Some of our most-requested features have been newly-added: Joins, Query Quotas, and an update to our visualization engine.

Joins

With the new join() filter, Humio makes it easy to update a dashboard or perform a single search with combined data from one or more repositories. For example, a result may combine data from one set of records showing IP addresses that accessed the system at a certain time, and another set of records with IP addresses assigned to employee names.

join({subquery}, field=arg1, key=arg2, repo=arg3, mode=inner|left)

See the Query Function reference documentation on join for more details.

Humio provides two join modes: inner joins and left joins.

Inner joins return the intersection of matching values between the primary query and the subquery. For example, for the two data sets above, it would returns a list of IP addresses that accessed the system that were also assigned to employees, along with the employee name.

Left joins return everything in the primary result, along with everything that matches the subquery. In this example, it would lists every IP address that accessed the system during a certain period (whether an employee matched the IP address or not) — plus the employee name listed for those that with a matching IP address.

Joins are one of those features that a lot of our customers have been looking forward to, especially within the security space. When you work with threat feeds, you want to join a stream of malicious IP addresses with your firewall logs, and get a warning if there’s something wrong. You want to enrich your logs with Active Directory — if you have dynamic IP addresses, you can look one up in the AD and see what machine it is.

Anders Jensen
Humio VP of Engineering

Humio joins combining geoip_dst.ip from two datasets

Humio joins combining geoip_dst.ip from two datasets

In addition to joining two queries against a single repository, the join() function can also be used to return a result set from more than two repositories.

Joins are such a requested feature that we’ve been able to work with users as we built it — we haven’t dreamed up the requirements ourselves. We’ve had a lot of scenarios from different users, and we worked with them early on to solve their problems. It’s really nice to have this interaction with users and then build the feature that way.

Christian Hvitved, Humio Chief Engineer and Co-Founder

See more about using joins in the Humio Documentation: join search function.

Query Quotas

Query quotas can limit the amount of CPU, memory, and I/O resources available to any one user when searching. Usage is tracked continuously as queries are executed. Whenever a user exceeds their quota, the query is stopped and the user is notified. As long as the quota is exceeded, any new queries arewill be rejected as soon as they are initiated until they get an override.

Query quotas assignment screen

Query quotas assignment screen

Quotas can be specified in a number of different time intervals in order to allow the Cluster Administrator to protect against both short-term accidental heavy queries and longer-term general heavy queries. Default query quotas apply to all users of the Humio cluster. Individual users can be assigned unique quotas, or have default quotas overridden.

We have customers that are running a cluster with a lot of users, where somebody is running an API every second, or they are making a huge query — doing something they probably shouldn’t. The quotas will kick in here, and Humio will just pause those things.

We’ve also put a lot of work into improving the query monitor. If you’re a cluster admin, you can go in and see what is taking up resources, what people are doing — and you can kill queries so there are resources for everyone.

Christian Hvitved
Humio Chief Engineer and Co-Founder

Cost points are the unit Humio uses to schedule, limit, and monitor queries. A cost point is a combination of both the memory and CPU consumption that a query has, and can be used as a measurement of how expensive a query is overall. A static cost point quota measures cost points spent on processing historic data. A live cost point quota measures how expensive it is to run live updates for the query.

Query count can also be used to limit the number of queries that can be executed within each time interval. This can be useful if you find that users are executing too many queries, as there is an inherent cost to starting and coordinating a query that isn’t included in cost points.

If you have a lot of users, and you want to let them go as they get more comfortable and knowledgeable about Humio, you can lift the quotas.

Anders Jensen, Humio VP of Engineering

Read more about query quotas in the Humio Documentation: Query Quotas.

Support for VegaJS visualization language

We transitioned the Humio chart engine to Vega and converted existing charts. We will be opening up for users to define custom visualizations in the future.

Vega is a data visualization library that has gained a lot of popularity recently with over 8k stars on GitHub. What makes Vega different from traditional charting libraries is that its charts are written as declarative JSON specifications. That means that it is not an API for drawing, but rather a language for defining the connections between the data and the dots, lines, and boxes that make up a visualization. Since Vega's specifications are declarative, we don't have to worry about all the issues a JavaScript API would pose — while still giving the plugin author a full set of charting tools.

Humio's sheer search and processing power can reduce a massive data set and pass it on to Vega where it can be filtered, transformed and explored in a dynamic and interactive way. A match made in Heaven. … This is the first step in improving our visualization and drill-down capabilities.

Thomas Anagrius
Humio Lead Frontend Developer

Because we added Vega into Humio under the hood with the built-in set of visualizations, you won't have to learn Vega to use Humio. We will open up for partners and the community to share their own Humio-compatible Vega visualizations in the future.

Replication of Humio chart types using Vega

Replication of Humio chart types using Vega

Query rollups into “Other”

Additional UI improvements

There are a number of UI improvements, back-end improvements, and bug fixes. Here are some highlights. All the improvements along with dozens of bug fixes are listed in Humio Documentation.

  • New Queries drop-down to search saved and recent queries: Queries page has been replaced. (1.7.0)

  • Query errors highlighted as you type in the search page. (1.7.0)

  • Improved Query Monitor makes it much easier to find expensive queries. Query Monitor. (1.7.0)

  • Pass URL parameters to Humio dashboards to set the dashboard’s global time interval, so it’s easier to integrate with external systems.

  • Disable shared dashboards completely if your organization has strict security policies.

  • Load a specific time window when launching a dashboard. (1.7.0)

  • Improved word-wrap. (1.7.0)

  • Autosize columns in the event list will adapt to the screen size when word-wrap is enabled. (1.7.0)

  • Sticky word-wrap and event list orientation, so revisiting the search page will keep the previously selected options. (1.7.0)

  • Disable automatically searching when entering a repository search page on a per-repo basis. (1.7.0)

  • Dashboard time selector panning and zooming — like the one on the search page. (1.7.0)

  • New function callFunction, allows you to call a humio function by name. (1.7.0)

  • New function json:prettyPrint() and xml:prettyPrint(). (1.7.0)

  • New function top that’s more efficient: e.g., top(field, max=value, limit=5). (1.7.0)

  • Percentile function faster and more precise. (1.7.0)

  • Sticky autosharding set by user on a specific (input) datasource. (1.7.0)

  • Allow explicit auto as a span parameter argument in bucket and timechart, so it’s easier to set span from a macro argument. (1.7.1)

  • Remove 64 K restriction on individual fields to be parsed. (1.7.1)

  • New Jar Utility: Usage allows decrypting a file that was uploaded using bucket storage outside the system. (1.7.2)

  • New LOG4J_CONFIGURATION allows a custom log4j file. (1.7.2)

  • Webhook notifiers optionally not required to validate certificates. (1.7.4)

  • Chromium added to the list of compatible browsers. (1.7.4)

Query Dropdown and Save Search

Query Dropdown and Save Search

For additional information, see the Humio documentation for Humio 1.8.0.

Stay informed

Keep in touch about what’s happening with Humio releases by joining our Slack community. Our product engineers and support experts are available to answer questions and remove any roadblocks.

We also have an RSS feed for Humio releases: https://docs.humio.com/release-notes/index.xml.

We invite you to find out more about Humio by scheduling a live demo with one of our product experts. Or, get started with a free 30-day trial.

Start your free trial now, available Self-hosted and SaaS, or request a demo.