How-To Guide

Make networks and data more resilient and secure in higher education

Contents
How-To Guide

Make networks and data more resilient and secure in higher education

Higher Education system administrators face challenges keeping up with an ever-expanding IT landscape, relentless cyberattacks, and the growing needs of students, faculty, and staff.

IT and Security professionals are responsible for understanding how the entire system is working, to keep their system resilient and safe, and to be able to solve problems as they arise. This requires maintaining observability of devices across the system, and the ability to quickly understand the root cause of issues or security incidents to be able to prevent them going forward. This presents a significant monitoring and security challenge, especially with distributed hybrid environments, bring-your-own-device policies, and layers of different types and different generations of software.

Student engagement and academic success relies heavily on a variety of systems, from legacy software to the latest technology. With that comes a growing number of servers, systems, networks, applications, and endpoints, resulting in an exponentially-growing volume of data.

Recently, schools have moved from conducting a few classes online to conducting 100% of classes and nearly all school business remotely — requiring the deployment of new technology while rapidly expanding the number of remote connections to a rapidly growing and increasingly vulnerable network surface.

And as a backdrop to all of this, budgets are declining, constricting the availability to install new systems and maintain existing systems, while the requirements to stay current with technology never end.

Deploying modern log management technology has the potential to immediately start addressing the complexity of a growing Higher Education network.

THE HIGHER EDUCATION IT ENVIRONMENT

Campus data diagram

Section icon for: Modern Log Management monitors for performance and security issuesModern Log Management monitors for performance and security issues

Modern log management provides a solution for the concerns of IT and security administrators in these complicated higher education settings. Because nearly everything connected to the network produces log data, it’s possible to collect and monitor it in a way that helps illustrate the performance and health of everything in the IT environment. And as we’ll show later, it does it all in a way that can generate business value while reducing already-limited budgets.

Having access to all this data can provide significant business value. Educause recently published EDUCAUSE 2020 Top 10 IT Issues, where they discuss how data can help the school understand and meet the expectations of students.

“Thanks to the rapid evolution and adoption of technology, higher education institutions have an abundance of data at their disposal—data that can be analyzed to better understand students. This data analysis, comprehension, and utilization is a must for 2020; with real insight into students' tendencies, goals, and habits, faculty and staff can communicate with individuals more seamlessly and can potentially intercept trajectories if students are not on the path to success.” 1

Many universities and colleges already collect data for IT Operations. The research firm Gartner shares insight into other ways to think about using log management. They point out that “many clients are unaware that log management tools can be leveraged for use cases beyond just collecting logs in a central repository to have them available for after-event analysis.” 1

“Use a central log management capability and tool when there are budget and staff constraints, basic security monitoring requirements, and compliance-specific use cases.” 2

Gartner describes how modern log management is used for more than IT Operations. They point out several additional security and compliance use cases emerging from using Central Log Management (CLM).

How organizations leverage CLM

Organizations are using log management to collect, monitor, and store streaming log and event data, and use it to identify and track down the root causes of performance problems, outages, delays, and data privacy and security incidents. Log management can significantly boost the abilities of higher educational institutions by providing the ability to:

  • Achieve complete observability by monitoring IT Operations to track the performance of servers, networks, devices, and endpoints.

  • Monitor for external and internal security threats with real-time alerts.

  • Conduct comprehensive investigations using correlated events to discover system problems, app performance issues, and the root causes of security incidents.

  • Remain compliant with government and industry regulations by retaining data and storing required access or change data.

  • Prepare for the unknown by collecting data that could be useful when an unanticipated problem arises.

Observability in higher education networks

In the complex IT environments of higher education, the only way to know if your environment is performing as expected is when its components are observable.

A device or event is observable when its activity can be observed. In a general sense, to observe something is to watch it carefully with the hope of arriving at a judgment. In physics, something is observable if it has a property that can be measured directly, like temperature or position.

Observability is when you’re able to understand a system from the data it provides, and you can explore that data to answer any question about what happened and why. Observability is vital to keeping the complex environments required in higher education healthy and secure.
Geeta Schmidt

Humio CEO

In the complex IT environments of higher education, the only way to know if your environment is performing as expected is when its components are observable.

A device or event is observable when its activity can be observed. In a general sense, to observe something is to watch it carefully with the hope of arriving at a judgment. In physics, something is observable if it has a property that can be measured directly, like temperature or position.

In a system, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. To be truly useful, those measurements should help pinpoint problems, and help the observer arrive at what happened and why.3

If the objective of a System Administrator is to understand the health of the system and keep it running efficiently and securely, the goal of observability should be at the forefront.

Here are a few resources that explain more about observability:

Section icon for: Challenges with funding in Higher EducationChallenges with funding in Higher Education

Anyone working at a publicly-funded higher education institution doesn’t need to be reminded that budgets are always a challenge. Even before the recent challenges from the current pandemic, funding technology in higher education has always been a significant challenge.

With falling student enrollments, increases in competition, decreases in state funding, and overall revenue shortfalls, organizations across campus are faced with decreased budgets. IT Administrators are challenged to manage improvements to overall systems while reducing the time spent maintaining them.

We’re in an area where budgets are tightening, and it’s going to be more difficult to get the resources we need. So I look for innovations. I look for technologies that deliver what we need at a low cost. Not just costs from the technology itself, but lifecycle costs, implementation, and other resources to manage and run the operation.
Miguel Adams

Security Engineer, Government Agency

Fortunately, advancements in technology often come with efficiencies that are passed along to the customer. This is true for consumer electronics, and it is true for software and systems.

Disruptive technology challenges traditional leaders with innovation and lower technology costs

Innovative companies are creating new solutions that are technically advanced, predictable, scalable, and cost less. For example, modern log management has advanced beyond traditional database technology by eliminating storage-bloating indexes and compressing log data by up to 10-20x or more. This can dramatically reduce the number of servers and storage devices currently used by universities and colleges.

Newer technology tends to be easier to learn and use

Development teams that use approaches like Agile and DevOps focus on efficiency, flexibility, and most importantly developing for the customer experience. This leads to technology that is easy to learn, and intuitive to use. With this in mind, applications are much easier to use, and platforms and systems require less training and fewer resources to keep things running.

We spent numerous hours a week managing our old system. Now I can get at least the same amount for an hour a week or less — and we spend an hour a week because it is such a useful tool.
Rick Miller

Director of Systems Administration, Kutztown University

Because of this user-centered approach and the benefits that come from process improvement, new technology often makes things simple that were once complicated. Because there are fewer resources available in higher education, the simplification of administrative tasks can reduce redundancy, take steps out of processes, and improve the results from a product or service.

Subscription-based licensing limits costs to what is actually being used

With the move to the Cloud, universities and colleges are seeing cost savings that come from capital costs reduction from spending on equipment, infrastructure, and dedicated software. By shifting those resources to the cloud, it allows schools to rent processing power without having to keep expensive hardware standing by. Most SaaS contracts include the costs of upgrades, new hardware and software, and more. There are few dedicated IT engineers needed, and there’s no energy consumption or other overhead needed to keep a datacenter running.

Cloud storage is reliable, fast, and inexpensive

Cloud providers offer durable storage that scales nearly infinitely. Data can be retrieved from the cloud nearly as quickly as it can from local disks, depending on the configuration. A single API integrates storage into applications, making storage less expensive and more convenient. The cost of using cloud object storage is remarkably inexpensive, especially if it’s not used often. For more information, see our How-To Guide: Optimize the stack with cloud storage.

As data grows, license limits force filtering logs

Traditional log management licenses are based on ingest volume — the amount of data that is collected and processed per day or per month. For the first year, costs may be in line with the amount of data. But as data volumes increase, or as new use cases are investigated, costs can go up dramatically. What seemed reasonable at 64 GB per day simply isn’t affordable at 128 GB per day.

When expensive volume-based pricing eclipses the original requirements, administrators are forced to find ways to limit the amount of data they collect. This undermines the goal of system observability, and invariably leads to slower response times, and decreased use of tools that are in place.

Unlimited licenses lift the constraints of limiting log management, and help make it affordable to log everything. In addition, this type of license could be considered as a central solution that could be used by all departments across the campus, helping reduce their overall costs and providing correlated data for improved analytics. For more information about unlimited log management, visit our website page: Why Unlimited.

This guide outlines how to use modern log management in a higher educational setting. It provides information on general concepts and offers resources for additional technical information. The following 5 steps offer a way to get started. There is more detailed information later on the page.

Steps to make networks and data more resilient and secure in higher education

Centralize data collection and access

Collect log and event data from servers, networks, devices, and users to make it easily searchable.

Read more about each step

Section icon for: Log data sourcesLog data sources

There are endless logs, events, and other machine data that are available to collect. Depending on the infrastructure or applications that Higher Education system administrators are responsible for monitoring, there are dozens — perhaps hundreds — of sources of data. The logs created from these sources were designed for IT and Security teams to do performance management, threat detection, and conduct troubleshooting.

  • Applicant data

  • Student data

  • Faculty and staff data

  • Financial data

  • Research projects

  • Marketing and outreach

  • Alumni engagement

  • Asset tagging

  • Student application, registration, grades

  • Parking, meals, lab fees

  • Bookstore

  • Learning Management Systems

  • Legacy applications

  • System event logs

  • Email transaction logs

  • Windows, Linux, UNIX servers

  • Web servers

  • Databases

  • Cloud servers, containers, hypervisors

  • Storage, SAN

  • SaaS applications

  • Laptops or desktops

  • Mobile devices

  • Routers, switches

  • Controllers, access points

  • Load balancers

  • Proxy servers

  • DNS logs

  • Telecomms, call detail

  • Authentication: AD, LDAP

  • Antivirus data

  • Threat intelligence

  • Firewalls

  • SSL/TLS

  • X.509 certificate logs

  • VPN access

  • SSH/FTP access

  • IoT devices

  • Remote sensors

  • RFID

  • GPS location

  • Gym, dining hall, and laboratory access (biometrics)

Section icon for: ChallengesChallenges

Securing networks and devices and maintaining the health of the IT infrastructure in Higher Education is complicated. Managing existing services and investigating emerging technologies can become overwhelming due to a long list of challenges.

Growing data volumes make license costs unpredictable

As data loads increase, higher education organizations are discovering that their current log management tools are inadequate to meet their growth needs. As log volumes grow, so does financial pressure on security teams who want to log everything in their system but can’t afford it. Oftentimes the result is leaders choosing to limit which logs they capture, sacrificing the ability to see all events in their system, and also their ability to search historical data and find the answer to novel questions.

Specialized software, multiple types of OS and hardware, and massive amounts of data being used in non-standard ways

University and college networks are created to be open and accessible, but because of that, each student or faculty member ends up using their own systems and devices. While this can enable creativity and innovation, it makes managing IT and security operations increasingly difficult.

New students and devices every new school term

Every new quarter or semester, students enter and leave the school. New students bring new devices with them, so the landscape keeps changing. IT administrators can’t protect what the were protecting six months ago the same way.

Increasing regulatory compliance requirements

Student data contains nearly every kind of protected data, including financial data, health data, and personally identifiable information. Due to regulations and privacy policy, data must be encrypted, transmitted, and stored securely. Doing this responsibly requires extensive planning and detailed implementation. Doing it incorrectly exposes the school to penalties, lawsuits, compensation, remediation, and loss of revenue and reputation.

Higher Education is a valued target for cybersecurity attacks

School networks are often more open and less secure than business or government networks. They also store personal records and valuable intellectual property from sponsored research. Higher education networks are favored targets because of the value of the records they hold, and because they can be easier to access.

Isolated views of security data

In many organizations, IT monitoring and security solutions end up becoming isolated, and used by different organizations across the institution. As IT operations, Security, and Application Development teams grow, they can drift further from each other. By combining infrastructure, operations, development, and security data together with a central log management platform, all teams can get closer to the entire range of streaming data from across the system.

Installing and maintaining systems requires dedicated resources

Open-source systems are widely used in higher education. While many open-source tools offer similar benefits of licensed software, they often come with additional complexity that requires dedicated resources for installation and maintenance. Closed-source systems often require less labor, but they may still require one or more dedicated staff to provide updates and daily maintenance.

Modern systems are purpose-built for efficiency and ease of use, so they usually require far less maintenance. This can greatly reduce operational costs, and free up IT staff resources for mission-critical projects.

Delayed data prevents real-time detection

Many systems rely on data that needs to be indexed and stored before it’s available for updating dashboards or delivering alerts, delaying the time to detection and resolution by several minutes. Modern log management streams real-time data without indexing, so alerts are updated in real time, and investigation can begin the moment an incident occurs.

Section icon for: Choose the right log management solutionChoose the right log management solution

There are answers to be found in the data that log management collects, but it can be challenging to manage the data and connect the information in a way that correlates the data and makes it easy to search.

Modern log management can make it painless to collect data from across the entire IT environment, bring it all together, and combine it in a way to answer any question about what’s happening.

For the best results, educational institutions should use a modern log management solution optimized for speed and efficiency. Look for these hallmarks to find the best high-throughput, low-cost system.

CHECKLIST

Invest in modern log management tools that are fast, flexible, efficient, and easy to use.

  • Capacity to ingest and store all data required

  • Fast search with near-zero latency from ingest to being searchable

  • Streaming data ingest

  • Index-free technology for real-time ingest, free-text searches, and optimized storage

  • Architecture for speed, efficiency, and flexibility

  • Affordable license fees that scale predictably as data requirements grow

  • Easy-to-use free-text search

  • Data enrichment to augment raw data, including joins from multiple data sources

  • Dashboards updated in real time

  • Flexible visualization capabilities

  • Data compression for efficient storage and data transfer

  • Long-term retention and storage using inexpensive cloud storage

  • Resilient design that doesn’t require extensive ongoing maintenance

  • Enterprise-level security

  • Self-hosted or SaaS

5 Steps to make networks and data more resilient and secure in higher education

Most universities are using traditional log management systems and security monitoring systems that they originally licensed years ago. While it may be tempting to continue using tools that are familiar, there are opportunity costs of not looking at systems built recently to take advantage of today’s evolving environments.

Start with one use case, optimize it, and build the system for additional use cases. This approach allows a more thorough investigation of available technology, which will demonstrate the full power, speed, and usability of modern advances in log management platforms. In the end, it will undoubtedly make the initial time investment worthwhile.

The following five steps can provide a framework for implementing or enhancing IT Operations with a modern log management system.

The opportunities in higher education for consolidating resources and centralizing data management are endless. Unfortunately, many schools have a culture of siloed operations. While this makes it easier to create and implement technology, it can lead to inefficiencies and missing out on the benefits of scale.

Schools are encouraged to approach IT more holistically, and they’re beginning to move from having siloed applications and IT operations that are managed by different departments across the campus. By beginning the process of centralizing systems and operations, each team can benefit from the cost savings coming from scalability, and every group can redirect resources to fulfilling the mission of their organization.

One place to start is in a place where traditional systems aren’t providing the value they were originally purchased to offer. In some cases, this takes years of investigation and migration. In others, it’s as easy as installing a new service. Modern log management is one of the platforms that can be installed quickly and deployed alongside existing systems as a trial or proof of concept. Because log management can begin creating value with one data source and scale to hundreds of terabytes a day, it’s one of the places that’s worth spending a little time investigating.

Log management can accept a high throughput of data — from servers, apps, network devices, customer devices, and almost anything that performs a function — and keep it all in one central location. Even though the data is varied and unstructured, it can still be available to search in real time without indexes. This makes it easier for users across the organization to explore everything they have permissions for, and even join search results from several different data sources, getting results in seconds. In addition to cost savings, this can lead to insights into operational and strategic initiatives that were unavailable before, and can uncover strengths and pain points across the whole system.

Consider the log and event data being collected and used across the institution. Reach out to other organizations that manage these data sources, and discuss use cases for collecting it centrally and making it available for analysis.

Get started with modern log management

It’s easier than you might think to get started. You can get a good sense of the ease of operation and the benefits of index-free, streaming observability in a matter of minutes, and at no cost.

  1. Start with a 30-day trial
    Deploy a free 30-day trial version of our log management system on your own servers or private cloud, or install it as a SaaS.

  2. Learn the basics
    Take advantage of an interactive in-app tutorial that continuously streams simulated log data from a web server. It will step you through the basics of searching and visualizing the data.

  3. Prepare a repository to ingest your own data
    Create a data repository locally, or use a cloud sandbox.

  4. Assign a data parser
    Use one of the built-in parsers or define your own. Built-in parsers are available for most popular logging formats (AccessLog, Syslog, JSON, Zeek), and can extract almost anything with the key value (kv) parser. You can also create a custom parser.

  5. Ship your school’s data to the system
    For logs, it is easy to start with the widely-used open-source Filebeat shipper. For metrics, start with Metricbeat. Both are very popular and widely supported tools.

    Configure a data shipper (Rsyslog, Filebeat, or Logstash)
    Integrate with a platform (Kubernetes, Docker, or DC/OS)
    Ingest through Humio’s REST API


For more information, watch an on-demand recording of the Humio Quick Start Workshop, or visit the Humio Documentation site: Getting started with Humio.

Improve Network Monitoring

The networks of Higher Education institutions can generate hundreds of terabytes of data a day. Network sensor appliances can automatically collect and compress the data from data-intensive networks. They add traffic logging, file extraction, analysis automation, and they provide valuable information not found in network or Netflow logs.

  • Asset tagging by Campus IP

  • Geo IP/ASN for external IP addresses

  • Whois information for domains

  • Inbound/outbound ratio

  • Hundreds of fields of data about dozens of protocols.

  • Precisely time-stamped and interlinked data files

  • Logs that are 1/100th PCAP’s size

Learn more about how network monitoring appliances improve observability from these on-demand webinars:

Even though the systems of higher education institutions generate dozens of terabytes of log and event data every day, modern log management can collect massive amounts of streaming data, and make it available to visualize, alert, and search the moment it arrives. Use this streaming data to understand what’s happening across the system, and look for ways to improve performance, reduce risk, and enhance the user experience.

Once the logs are shipping into the system, it will be easy to see what is happening by searching it. Queries are the foundation for dashboards, alerts, or to learn more about a specific incident. How hard is it to learn a new set of commands and queries? Not hard at all! Here are some basics to illustrate how easy it is to get started.

Get started creating queries

If you’re familiar with SQL (structured query language), grep (global regular expression print), or regex (regular expressions), and have ever used pipes ( | ) or logical operators (and, or, not, and !), you have most of what you need to find and visualize anything contained within the log data.

#host=github #parser=json | // <-- Tag Filters
repo.name=docker/* | // <-- Filter Expression
groupBy(repo.name, function=count()) | sort() // <-- Aggregates

Transformational Queries (also called Filter Expressions) filter input or add/remove/modify fields on each event. These include filter expressions like:

name = "Peter" and age > 25
color := "blue"

Aggregation Queries (also called Aggregation expressions) are function calls. These functions can combine their input into new structures, or emit new events into the output stream. A query becomes an aggregation query if it uses at least one aggregate function like sum, count, or avg. For example, the query “count()” takes a stream of events as its input, and produces a single record containing a _count field.

loglevel = ERROR | timechart()
x := y * 2 | bucket(function=sum(x))

Joins. The “join()” function returns a combined result set from two queries.In addition to joining two queries against a single repository, the join() function can also be used to return a result set from more than two repositories. To do so, use the optional repo parameter to define the repository (as arg3) you want the subquery to run against. Note that the user running the join() query must have permissions on both repositories.

join({subquery}, field=arg1, key=arg2, repo=arg3, mode=inner|left)
#type=accesslog statuscode>=400

| join({useragent != ""}, field=sourceipaddress, key=browseripaddress, repo=weblogs)

To learn more about the flexibility and power of modern log management queries, see Humio Documentation: Query Language.

Live Queries

Live queries provide a way to run searches that are continuously updated as new events arrive. Live queries use streaming data to update the results of queries in real time. They are important for updating dashboards, alerts, and running webhooks the moment data arrives.

In a live query, the time interval is a time window relative to ‘now’, such as ‘the last 5 minutes’ or ‘the last day’.

A live query will start doing a historical query as well as setting up the live streaming part. New events are pushed into the streaming query.

Case Study

Providing observability and cutting labor costs to nearly zero at Kutztown University of Pennsylvania

Kutztown University of Pennsylvania is a public university in rural Eastern Pennsylvania 50 miles outside of Philadelphia with approximately 7,000 undergraduates and 1,000 postgraduate students. Their centralized IT Department of about 35 employees is responsible for the upkeep of the university’s networks and systems of enterprise desktop computers.

We’ve had to tighten our belts so many times. We have to be very cautious and think forward about any purchases that we make.
Rick Miller

Director of Systems Administration, Kutztown University

Several years ago, they were to start monitoring and retaining log data by state auditors. They purchased a Security Information and Event Management (SIEM) tool, and configured it to be compliant. Although the system was operational, they found that the hours it would require to keep it updated or make any changes was too prohibitive. As a department with limited resources, they didn’t have the time needed from engineers to keep the SIEM working efficiently, and they were seeing diminishing returns.

Our previous system required so much effort and handholding and management that it ended up not doing anything for us. We found it to be cumbersome enough that we just didn't have the time to do it.
Rick Miller

Director of Systems Administration, Kutztown University

To have log management that worked for them, they needed a system that would collect all logs, comply with state recommendations for data retention, provide some security benefit, not break the bank, and most importantly – not create a workload on its own.

The IT department at Kutztown found a modern, index-free log management system that was flexible and powerful enough to meet their use cases and also stay under budget for the foreseeable future. With minimal effort, the new system was answering security questions about where attacks were coming from and detecting sources of activity that were weighing down the system.

This enabled the IT department at Kutztown to increase observability without having to dedicate staff to it, and changed the way Kutztown interacts with their logs. Log management has gone from an underused, inaccessible, background part of their data to being a front-and-center primary source of information.

Read the Customer Case Study >>

Once the data has been collected, it can be searched and analyzed to answer questions about system and security issues. Perhaps more importantly, it can be used to answer questions about what’s happening with the operations of the school and with the performance of the students.

Data Analytics Can Save Higher Education

The Association for Institutional Research, EDUCAUSE, and the National Association of College and University Business Officers recently published a joint statement to strongly argue that data analytics “can save higher education.”

“We strongly believe that using data to better understand our students and our own operations paves the way to developing new, innovative approaches for improved student recruiting, better student outcomes, greater institutional efficiency and cost-containment, and much more. Data is an institutional strategic asset and should be used as such.

“For every year we fail to use data effectively to improve operations or to make better financial and business decisions, we threaten the financial sustainability of our institutions.” 4

Meet student needs

The success of students relies on all the resources of the college or university. They need to be engaged, and they need help making progress toward graduation and careers. Data from across the campus can offer insights into the impact of classes, student resources, activities, faculty, facilities, research programs, and more.

Analyzing data can improve university operations in countless ways, including:

  • Confirming that course offerings appeal to students

  • Matching demand for courses with faculty and facilities

  • Ensuring that students have the credits they need to graduate on time

  • Making efficient use of computing resources, parking, and other facilities

  • Identifying which students are on track, which are close to graduating, and which are eligible to be auto-graduated

  • Helping students that are having problems and determine strategies and interventions to help them

  • Measuring the outcomes of programs to ensure goals are being met

Learn how students learn

Online and software-based learning platforms are being used more frequently in higher education. This movement has resulted in an explosion of data, which can now be used to improve educational effectiveness and support basic research on learning. Learning analytics is a powerful new technique to improve learning-at-scale and student modeling that drives intervention and improvement in educational software and systems.

With new technology in education comes new approaches for helping students. Advising staff can be more effective if they have early alerts to warning signs, and instant access to student information they can use in their outreach. Data dashboards can help illustrate problem areas in real time to highlight students who show patterns that they are struggling.

EdTech Magazine recently highlighted work being done by Georgia State University to tap into GSU’s student information systems for data on its 53,000-plus students. For example, the platform might alert an adviser that “Sue Jones just failed a math quiz” so the advisor can recommend that the student attends a free tutoring course. GSU also developed an in-house system to track class attendance by monitoring logons to the wifi and learning management system. With that information, they can predict demand for certain courses so the university has enough seats are available. 5

Get started sharing data with live dashboards

Once it’s determined how data can answer business questions, custom dashboards can be created to share relevant system data with organization leaders, and other functional groups.

Dashboards can display a collection of different widgets, each with their own set of queries, in a browser typically used on a passive monitor without requiring authenticating as a user. The URL in itself is then accepted as authentication and when accessed in that manner the dashboard is not interactive: You can only execute the exact queries that are saved on the shared dashboard.

A dashboard can be made up of anything you can make a query for, but usually consists of a number of time charts and tables all based on “live queries.” If you add a non-live query to a dashboard, the widget will not get updated.

Create new dashboards

Creating a new dashboard is as simple as saving a search and choosing a style of chart. Once it’s placed on the page, it can be moved, adjusted, and customized.

Share dashboards

A read-only dashboard can be shared by creating a share link. Share links have names to help you remember where they are being put to use. The shared dashboard always shows the latest version because it’s based on live queries. If you share the dashboard, then later edit the widgets in the dashboard, the read-only share links being shown on other screens will show the latest version of the dashboard.

Administrators can control which departments see what data in safe-to-share, append-only data flows that cannot be erased. Learn more about how Role Based Access Controls (RBAC) can be managed from Humio Docs: Role Based Authorization.

Continually adjust the dashboard to make it work for the organization. Make sure it illustrates the most important information, and that it helps the business prioritize action. It will be useful as long as the data being reported provides value, and that it answers relevant questions for the audience.

For more information on creating and modifying dashboards, visit Humio Docs: Dashboards.

Case Study

EDUCAUSE shares a use case at a multi-campus college specializing in healthcare education. They used college student data to create a predictive model that could help pinpoint signs of at-risk students; to make outreach techniques more effective; to make adjustments to the curriculum; and to give practical advice to students to help them succeed.

By analyzing data from a courseware system and the college’s student information systems, they identify two metrics:

  • Low percentage submission - LPS tracks students who fail to submit all of their assignments during the first two weeks of class.

  • Procrastination index - PI shows how often a student starts assignments in the last quartile of time before the due date.

The school was able to spot students who are at risk of failing:

  • Only 25% of LPS students ended up passing their course

  • 70% of students who had a high PI ended up failing.

These data points helped the school perform effective outreach and take action that helped the students get back on track with tips that were informed by the research.1

Learn more from EDUCAUSE 2020 Top 10 IT Issues >>

Create a Wall Monitor Dashboard

To share important information with your organization, create a wall monitor dashboard. This is great to display important results and status for use in a meeting space or lobby.

You can grant read-only access to individual widgets or entire dashboards publicly, or to a limited group. You can use “Shared Secret URLs,” which contain a special authentication token that grants read-only access to anyone that has the link.

Colleges and Universities are a highly-valued target for cybersecurity attacks. They store personal records (financial records, health information, and other sensitive data), and valuable intellectual property from sponsored research. Their systems are more open than a company of the same size, because a lot of the work depends on collaboration and the free flow of information. Higher learning institutions are viewed as favored targets because they hold many of the same records as businesses or financial institutions, but they can be easier to access.

Universities deal with a variety of data. The crown jewels for a university is the data that it is the custodian of, and that data comes from the students. That data may be a student's personal reports. That data may be a student's health records. That data may be payments from credit cards. That data has to be protected.
Fatema Bannat Walla

University Security Administrator

Use centralized log management to make segmented networks observable.

Universities and colleges are challenged by having a variety of data and an ever-changing number of devices accessing the network. Data is often segmented into several separate networks, to keep data accessible and secure.

Use Log Management to make your SIEM more powerful and efficient.

Many higher education institutions have installed Security Information and Event Management (SIEM) monitoring solutions to handle the basics of security. SIEMs are a good solution if they are kept up to data and have the staff required to keep them maintained. However, they can focus on a limited number of data points, and fail to provide a full view of the network. Log management makes it affordable to monitor all endpoints and maintain 100% network visibility, and store that data longer, helping organizations comply with compliance regulations.

Educational institutions with a SIEM installed should consider the benefits and costs of installing a modern log management system to offload the burden of log aggregation and storage from the SIEM. This will enhance the ability to conduct a more comprehensive search with more data sources and longer-retained data.

All we ever sent before was the Windows logs. So now, we’re getting the Windows logs, Linux logs, DHCP logs, student information logs, and so forth. So we’re getting all that extra stuff that we never could before.
Rick Miller

Director of Systems Administration, Kutztown University

Split-second search results make modern log management tools rocket fuel for security operations center (SOC) purposes. A security response team is able to go from an overview dashboard to specific logs connected with threats in seconds. A team can even hunt and find insights from encrypted traffic with help from network monitoring tools such as Corelight.

There are several options for running log management in the same environment as a SIEM. Consider running it in addition to the SIEM, collecting logs from a data pipeline, collecting logs in a data lake, or forwarding logs from the SIEM to Log Management.

For additional ways to use Log Management together with a SIEM, see the How-To Guide: Use Log Management as the Foundation of the Security Stack.

Get started using log management to enhance cybersecurity

Collect threat intelligence

Threat intelligence is knowledge about threats based on evidence. It contains actionable information on how threats occur, the mechanisms used, context and implications, and specific advice to prevent harm. Threat intelligence can be used to help make decisions about preventing or responding to each threat.

There are several places to get updated threat intelligence, many of which are provided as a community service. Some of these are databases of indicators of compromise, which can be used to alert administrators to possible threats. Threat intelligence can be used to enhance log data. For example, by looking at new IP addresses and comparing them against a list of bad actors, you can set up an alert when an identified IP address accesses the system.

For a list of examples of open source threat intelligence, see the How-To Guide: Use Log Management as the Foundation of the Security Stack.

Create security dashboards

To make the data being collected in the log management system more useful for security, set up a dashboard to analyze information about each use case. Use queries to create visualizations that illustrate the results that support each use case.

Set up alerts

Create appropriate alerts to notify security personnel when high-profile threat indicators occur that need investigating. Creating alerts is as simple as saving or building a live query, selecting a notifier type, and defining the frequency.

You can think of Alerts as one of two types. Single events that affect one or more users’ experience with the product. Usually not something that should wake engineers up at night over, but could result in a ticket on your issue tracker. A faulty state is when one or more components have reached a bad state and are unable to function properly. This usually affects most users and is something that should wake engineers up at night.

Forward alerts with notifiers

A notifier is a module that sends notifications when alerts trigger. Notifiers can be sent several ways, including email, Postmark, Slack, OpsGenie, PagerDuty, VictorOps, or using webhooks. Custom notifiers can be created using webhooks, with custom headers and body information. Find out more at Humio Docs: Alerting.

Use webhooks to take immediate action

Webhooks are the most flexible type of notifier. The webhook notifier can perform an HTTP(S) request to any URL, so it can be used to integrate third-party services that aren’t natively integrated. These simple web services listen for requests and execute scripts on the target system. The webhook agent and listen for the notification. This is a simple program designed to run on the host and advertise RESTful HTTP endpoints that when triggered via HTTP/GET or POST will execute scripts.

Don’t forget to write more log messages from these scripts and feed those back into the Log Management system as well, so you can be sure everything is working as intended. For an example of setting up webhooks to run a script when an alert triggers, see Humio Docs: Run Shell Scripts using Webhooks as Notifiers. Find out more at Humio Docs: Webhooks.

Case Study

Real-Time Observability to Identify and Investigate Security Threats at Michigan State University

Michigan State University (MSU) is a 160-year-old American University in Lansing, Michigan with over 50,000 students from all over the world. The logistical and security needs of the IT team at a sprawling campus in the US Midwest would stagger many corporations, even those in the world of tech.

Security information and event management (SIEM) was a key component to the success of the team at MSU, but with the existing setup in the market-leading SIEM system, they were only able to ingest 500 GB a day. They needed flexibility, security, and the ability to monitor their systems without a great deal of overhead. Additionally MSU needed to grow the amount of data they were bringing in and acting upon.

We consistently ran into licensing issues and volume pricing discounts were insufficient with our previous solution, so we actively reviewed products in the SIEM/log aggregation space and determined a modern log management solution was the ideal replacement.
Chief Information Officer

Michigan State University

MSU actively reviewed products in the SIEM and log aggregation space, and found a modern, purpose-built solution that met all of their core criteria. They were able to successfully replicate the work they had in the previous SIEM with a new log management solution. They were able to fully deploy the new system and completely replace the previous solution in under two weeks.

With an unlimited license, we will no longer have holes in our aggregate data. On our previous platform, we would have a large portion of data in the search results, but ultimately would still need to log into source systems and review logs manually.
Chief Information Officer

Michigan State University

With a new unlimited ingest pricing model, MSU now takes in more data with live insights and alerts, which enables deeper investigations into threats or issues when needed.

Read the case study >>

Educational institutions have a unique requirement of handling nearly all types of sensitive data. Administrators must stay informed about what is required for all types of data that are collected and stored. In many cases, there are specific requirements for creating policies for data governance, keeping data secure, protecting consumer data, and retaining records of compliance for auditing.

Schools need to ensure that they are complying with every data security and privacy regulation.

  • Implement processes to ensure transparency and control of all regulated data.

  • Consider how regulators and partners can be shown that the data collected and stored meets regulatory requirements.

  • Work with faculty to make sure that students, faculty, and staff understand the types of data collected and how it will be used.

  • Consider the regulations listed below, and use resources like the Higher Education Compliance Alliance Matrix for a more comprehensive view of national and local regulations.

Safeguard privacy rights of students, faculty, staff, and other constituents

Today’s higher education organizations are expected to protect sensitive student data–even as data volumes grow and compliance with federal, national, and state privacy laws becomes more complex.

The Family Educational Rights and Privacy Act (FERPA) is a US federal law that protects the privacy of student records through the regulation of their disclosure. It prohibits educational institutions from disclosing personally identifiable information from education records without the written consent of the parent or eligible student. FERPA applies to primary and secondary schools, colleges and universities, vocational colleges, and state and local educational agencies that receive funding under any program administered by the U.S. Department of Education.

While FERPA does not specifically require formal audits, data administrators should put auditing capabilities in place to allow security analysts to examine detailed activity logs or reports to see who had access, IP address entry, what data was accessed, etc. This data may then be tracked, logged, and stored in a central location in compliance with an educational institution’s data retention policy.

Below are several examples of regulations that may impact data collection and storage in higher educational institutions.

EDUCAUSE is a nonprofit association that helps higher education elevate the impact of IT. They have resources to identify laws, regulations, and policies for educational institutions. For additional guidance and helpful information, confer with your legal and/or audit departments, and review the Higher Education Compliance Alliance Matrix and the EDUCAUSE Library Compliance page.

EDUCAUSE shares common US federal data protection laws, state laws, and European laws that may impact higher educational institutions:

  • The Family Educational Rights and Privacy Act of 1974 (FERPA): Protects students and their families by ensuring the privacy of student educational records. Educational records are agency or institution-maintained records containing personally identifiable student and educational data.

  • The Health Insurance Portability and Accountability Act of 1996 (HIPAA): Requires medical and health insurance providers to protect the security and privacy of health records. This applies to student data when institutions have a campus medical center and student medical records are integrated with student educational records (which are protected under FERPA).

  • The Gramm Leach Bliley Act (GLBA): This law applies to how institutions collect, store, and use financial records (like records regarding student tuition payments and/or financial aid) containing personally identifiable information.

  • The Fair and Accurate Credit Transaction Act of 2003 (FACTA): Requires entities to be aware of the warning signs of identity theft and to take steps to respond to suspected incidents. This law applies to how institutions collect, store, and use student financial records.

  • Payment Card Industry Data Security Standard (PCI DSS): Includes requirements for security management, policies, procedures, network architecture, software design, and other critical protective measures. It requires collecting system and security logs, and has logging requirements. Specifies the audit trail history retention for at least one year, with a minimum of three months immediately available for analysis.

  • California Consumer Privacy Act (CCPA) is a California state statute intended to enhance privacy rights and consumer protection for residents of California. It regulates the collection, use, sale, and disclosure of personal information about California consumers and households.

  • General Data Protection Regulation (GDPR): Designed to consolidate data privacy laws across Europe, protect the data privacy of EU citizens, and reshape the way organizations approach data privacy. This approach treats the right to privacy as a fundamental human right. GDPR applies to entities with no physical EU presence if they control or process covered personal information of EU residents.

EDUCAUSE: US Federal Data Protection Laws

EDUCAUSE: EU General Data Protection Regulation (GDPR)

Get started setting up logs for audit and compliance

Audit log events can be marked as sensitive and non-sensitive. To make the audit trail trustworthy, sensitive actions are not mutable. Sensitive events include, assignment of roles to groups on repositories, changing retention settings on repositories, deleting repositories and data sources, and similar actions.

Retention settings for audit logs

Audit repositories have special retention rules that depend on the sensitive value.

Sensitive events logged. Sensitive logs are deleted by retention only when they are too old, controlled by a configuration option. Changing this setting requires a systems operator to change the configuration of the servers. Sensitive events include those that create, delete, or set retention on a repository; create, update, or delete a user; group membership change; role update or role change for a group; adding, removing, or changing ingest tokens; and more.

Non-sensitive events logged. Non-sensitive logs are deleted according to the regular retention settings for the repository. Examples include user sign in with Auth0 (logged only once); when the user signs in the first time and is assigned a local UUID; sign in with LDAP logs (stored every time the user verifies their username/password combination); UI query every time a query is submitted on behalf of the user through the UI; and API query submitted using the API-token of a user.

Other settings are available for log retention, access, and security. For more information, see Humio Documentation: Audit Logging

Deleting information for compliance

In some cases, you will need to delete individual event logs for GDPR compliance and privacy. Let’s say you need to remove data under the recently-passed GDPR laws to have all information on a user deleted from the database. Modern log management allows organizations to be compliant by being able to remove that data from the log.

The information that you want to delete is likely not just the FIRST_NAME and LAST_NAME columns in a relational database. It is scattered all over the place: in log statements, request logs, text messages, etc. It might be the person’s phone number that is mentioned in the middle of a text and not their unique user id. Find what you’re looking for by doing a blanket match against everything — any text and any field, both structured or unstructured.

(/john/i AND /doe/i) OR john@doe.com OR “+1 290 112 218” OR user_id=718

Note that delete_events is not a means for saving space or speeding up searches. It’s a tool to be deployed for exceptional cases, be they legal or technical, and depending on the segments to be revised, this can be a non-trivial operation. For delete_events to operate, a user must be authorized to initiate such changes. Learn when this is appropriate in the blog post: Delete Events and the Act of Forgetting.

Section icon for: Get started using HumioGet started using Humio

To understand what’s happening across complex environments, modern log management platforms like Humio provide visibility to streaming logs and event data. By monitoring this type of data the moment it happens, engineers, developers, and security professionals make sure their environment is healthy and performing as expected. When it’s not, they can search through the data to find out exactly what happened and prevent it from causing ongoing problems.

We invite you to see how Humio’s modern architecture redefines what is possible with log management.

Set up a Humio free 30-day trial. See for yourself how Humio can become the foundation for your security system. Our engineers are standing by to help you design a system that helps keep your organization more secure.

Find out more by visiting our website: Humio.com.

To get a free estimate of how much Humio can save your organization, visit our pricing guide.

Prepare for the unknown

With Humio’s Unlimited Plans, organizations are better prepared for the unknown. By logging everything, Humio can help customers uncover anomalies, threats, and problems as they happen, without worrying about filtering out data or fields in advance. Humio makes it easy to search months or even years of data to get to the root of what happened. Because Humio is index-free, every part of every log is available for searching.

Section icon for: About HumioAbout Humio

Humio's log management platform offers the lowest total cost of ownership, industry-leading unlimited plans, minimal maintenance and training costs, and remarkably low compute and storage requirements. Humio is the only log management solution that enables customers to log everything to answer anything in real time — at scale, self-hosted or in the cloud. Humio's modern, index-free architecture makes exploring and investigating all data blazing fast, even at scale. Founded in 2016, Humio is headquartered in London and backed by Accel and Dell Technologies Capital.

For more information, visit www.humio.com and follow @MeetHumio on Twitter.

Get alerted to new How To Guides or get a PDF of this one

Learn more from other How-To Guides

1. EDUCAUSE 2020 Top 10 IT Issues, February 10, 2020, by Carolyn Colman, Sarah Carey, Ling Chai and Andrew Sroka.

2. Gartner, Use Central Log Management for Security Operations Use Cases, Mar 20, 2020, Toby Bussa, Kelly Kavanagh, Mitchell Schneider (Gartner subscription required).

3. Humio, Observability (re)defined, December 30, 2019.

4. The Association for Institutional Research, EDUCAUSE, and the National Association of College and University Business Officers, Analytics Can Save Higher Education. Really, February 2020.

5. EdTech Magazine, Georgia State Tackles Racial Disparities with Data-Driven Academic Support, April 29, 2019, by Karen J. Bannan.

6. The Hechinger Report, Predictive analytics are boosting college graduation rates, but do they also invade privacy and reinforce racial inequities?, August 6, 2019, by Jill Barshay and Sasha Aslanian.