What’s New in Humio (September 2020)
New features include improved search speeds, free text search, and selfJoin and findTimestamp query functions.
September 3rd, 2020
The development team has been busy working on making Humio faster, more efficient, and easier to use. Additional information is always available in Humio Documentation: Releases. Details about version numbering for preview or stable releases are included at the end of this post.
Here is an overview of the features that were published recently.
Improved search speeds for many-core systems, free text search, load balancing of queries, TLS support, Iplocation database management changed
Improved search speeds for many-core systems: Improved query scheduling on machines with many cores. This can improve search speeds significantly.
Free text search: Free text search now searches all fields rather than only @rawstring.
Load balancing of queries: Humio can now balance and reuse existing queries internally in the cluster. The load balancer configuration to achieve this is no longer needed. See Humio Configuration and Reverse proxy configuration.
Added support for WebIdentityTokenCredentialsProvider on AWS.
Iplocation database management changed: The data source for the ipLocation() query function is no longer shipped with Humio but installed/updated separately. See IP location.
Prior releases include Export to bucket, findTimestamp, selfjoin, emergency user subsystem.
selfJoin query function allows selecting log lines that share an identifier; for which there exists (separate) log lines that match a certain filtering criteria; such as “all log lines with a given userid for which there exists a successful and an unsuccessful login”.
findTimestamp query function will try to find and parse timestamps in incoming data. The function should be used in parsers and supports the automatic detection of timestamps. It can be used instead of making regular expressions specifying where to find the timestamp and parsing it with parseTimestamp. Check out the documentation for details.
Export to bucket storage/S3: As an alternative to downloading streaming queries directly, Humio can now upload them to an S3 or GCS bucket from which the user can download the data. See docs.
Emergency user subsystem: If there are issues with the identity provider that Humio is configured to use, it might not be possible to log in to Humio. To mitigate this, Humio now provides emergency users that can be created locally within the Humio cluster. See docs.
Elastic Bulk API change may affect Fluent Bit: Fluent Bit users might need to change the Fluent Bit configuration. To ensure compatibility with the newest Beats clients, the Elastic Bulk API has been changed to always return the full set of status information for all operations, as it is done in the official Elastic API. This can however cause problems when using Fluent Bit to ingest data into Humio. Fluent Bit in default configuration uses a small buffer (4KB) for responses from the Elastic Bulk API, which causes problems when enough operations are bulked together. The response will then be larger than the response buffer as it contains the status for each individual operation. Make sure the response buffer is large enough, otherwise Fluent Bit will stop shipping data.
See: https://github.com/fluent/fluent-bit/issues/2156 and https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch.
For more information, see the individual release notes.
Several additional fixes have been applied in these releases. Please see the documentation for changelogs with details: Stable Releases.
The 15.0 preview release will be available shortly. Visit Preview Releases in Humio Documentation for more information.
More information on releases
We follow a versioning technique that is similar to the MAJOR.MINOR.PATCH numbering convention that’s used by Semantic Versioning. These are defined as:
MAJOR — Primarily for Marketing purposes, but would include high-profile changes.
MINOR — All new features or new functionality. Unlike Semantic Versioning, we allow backward-incompatible changes, such as changing compression algorithms.
PATCH — Only backward-compatible security or bug fixes are allowed. No new features or other, unrelated code changes are allowed. This is important, as it means that each patch release should be more stable than the last.
It’s worth noting that the criteria above primarily applies to stable releases. Preview releases will introduce changes typically expected in patch releases. See the Preview / Stable Releases section below for more information.
Preview and Stable Releases
Humio publishes new releases on a fairly quick schedule to get new features out to you as quickly as possible. This can sometimes result in some instability, so we have adopted an odd (preview)/even (stable) versioning system to better communicate how stable a release is expected to be.
As features are developed, an odd release series (where the minor version number is odd) will be available containing these new features. While these releases are intended to be usable, there is a chance that problems could arise. These will typically be as stable as releases prior to 1.7.0 have been historically. New features and other changes will be introduced as patch releases for any odd/preview release series. Once the features have been tested sufficiently, that preview series will be released as a new stable version.