Cybersecurity Log Collection with Graylog and Wazuh: Building a Robust & Open-Source Architecture
15 octobre 2023
Introduction
In this age of constant cyber threats, the meticulous management of system and application logs has emerged as an imperative.
Not only do these logs allow us to identify and respond to current security incidents, but they also provide a vital window into the past, enabling forensic investigations that can span back several months.
This paper presents an architecture for collecting and retaining logs, outlines the challenges in log processing, and provides comprehensive open-source solutions, such as Graylog and Wazuh (both running on the OpenSearch database engine), to address these issues. The bibliographic references are provided at the end.
Keywords: Logs, Monitoring, Active Response, Graylog, NXLog, Wazuh, OpenSearch
The significance of logs
Logs serve as the silent sentinels that monitor the activities of systems, applications, and users by carrying out the following functions:
- Detection of security incidents — Logs are the primary means by which security incidents are detected. They record events such as login attempts, system access, data transfers, and application activity. By carefully monitoring and analyzing them, security teams can identify unusual patterns, anomalies, and potential security breaches. In doing so, logs become the first line of defense in identifying and mitigating cyber threats.
- Security event correlation — Logs are not just standalone records; they can be correlated across various systems and applications to provide a holistic view of the security landscape. By combining logs from different sources, security teams can uncover hidden patterns, identify sophisticated attacks, and develop a more accurate understanding of the overall security posture.
- Compliance and auditing — Logs are essential for demonstrating compliance with various industry regulations and standards. Organizations in sectors like healthcare, finance, and government are often required to retain logs for extended periods to prove they are meeting data security and privacy requirements. Log data serves as evidence that an organization is following the necessary security protocols.
- Forensic analysis — Beyond their real-time utility, logs play a crucial role in forensic investigations. When a security incident occurs, forensic analysts rely on historical log data to reconstruct events, determine the scope of a breach, and understand the tactics, techniques, and procedures employed by attackers. This historical data can reach back several months, allowing for a comprehensive examination of the incident.
To sum up, logs are the lifeblood of cybersecurity, providing essential visibility, accountability, and historical context, ensuring that organizations can safeguard their digital assets and respond effectively to security challenges.
Challenges in log management
The collection and retention of logs pose several significant issues:
- Volume and variety of logs — One of the foremost challenges in log management is dealing with the sheer volume and diversity of logs generated by various systems, applications, and devices. As businesses expand and technology becomes more integrated into everyday operations, the quantity of log data can be overwhelming. Diverse log formats and sources further complicate this issue. To manage this, organizations require robust tools and strategies for log aggregation and normalization.
- Real-time analysis and correlation — Timely detection and response to cyber threats necessitate real-time log analysis. However, efficiently analyzing and correlating logs to identify patterns, anomalies, and potential threats is a complex task. Balancing the need for immediate alerting and response with the practical limitations of real-time monitoring is an ongoing challenge in cybersecurity.
- Log retention and scalability — Log retention policies must strike a balance between maintaining historical data for compliance, forensics, and threat analysis, and managing storage costs. As organizations generate vast quantities of log data, the cost and logistical complexity of retaining this data for extended periods become a significant issue.
- Log integrity, security and privacy — Log data often contains sensitive information, including user credentials and personal data. Ensuring that logs are adequately secured to prevent unauthorized access and protect privacy is an important concern. Regulatory frameworks impose specific requirements on the handling and storage of logs to safeguard user privacy.
Implementing an architecture with Graylog and Wazuh
In this section, we delve into the practical implementation of a log collection and retention architecture using Graylog and Wazuh.
We won’t provide here a detailed guide on how to install Graylog and Wazuh on the OpenSearch engine (which natively allows resiliency and scalability). Instead, we focus on the key steps and best practices for integrating these platforms to optimize log management.
Why combine Graylog and Wazuh?
Combining Graylog and Wazuh offers a synergistic solution for enhanced log management and cybersecurity. Graylog excels in log collection, analysis, and visualization, providing a user-friendly interface for monitoring digital activities. Wazuh, on the other hand, specializes in advanced threat detection and response.
By integrating these two platforms, you create a robust architecture that not only centralizes log data but also leverages Wazuh’s real-time threat detection capabilities. This combination offers a comprehensive solution for identifying and mitigating security incidents, making it an ideal choice for organizations looking to bolster their cyber defenses.
Real-world implementation
We have implemented Graylog and Wazuh within a lab composed of several physical servers (Linux and Windows), application servers (web, mail, Docker…), workstations (Linux and Windows) and two firewalls: an IPFire instance serving as the external firewall and a pfSense instance serving as the internal firewall. This configuration allows us to demonstrate how to manage a heterogeneous IT infrastructure.
Architectural principles
As shown in the figure below, we have opted to centralize Graylog within our collection system. This configuration allows Graylog to efficiently manage the collection of logs from various endpoints and to send alerts to Wazuh. While this is not the only possible setup, it lets us leverage Graylog’s full capabilities for log management while enabling Wazuh to focus on SIEM functions, specifically alert processing and active responses.
Graylog configuration
Graylog natively provides many capabilities for processing logs from devices using Syslog, and it can also ingest logs from specific devices such as Palo Alto firewalls. For Windows endpoints, an additional agent is needed — NXLog for instance (see below).
Graylog dashboards bring efficient monitoring of all incoming logs, and the inputs and streams features allow you to define specific ports, protocols, and indices (equivalent to database tables in OpenSearch terminology) for the ingestion of logs from each device or group of devices. For each index, log rotation can be specified (on a time period or on a file size criterion).
When it comes to Windows endpoints, NXLog agents offer a convenient solution to ensure compatibility with Event Monitoring, with a quick installation and a very simple configuration file. Example of an NXLog configuration file for a Windows endpoint:
<Extension gelf>
Module xm_gelf
</Extension>
<Input internal>
Module im_internal
</Input>
<Input eventlog>
Module im_msvistalog
</Input>
<Output graylog_udp>
Module om_udp
Host graylog.dudes.lan
Port 12200
OutputType GELF
</Output>
<Route 1>
Path eventlog, internal => graylog_udp
</Route>Triggering Wazuh
In order to relieve the Wazuh server, not all logs ingested by Graylog are forwarded to it: an alert mechanism, activated on a specific stream, notifies Wazuh only in the case of suspicious events (e.g. authentication errors).
Here is an example of event definition and Wazuh notification, through a dedicated API configured on the Wazuh server:
Wazuh configuration
The Graylog notification can be processed by Wazuh through an alerting rule (with alert level 11 in the example below) and associated with an active response, which necessarily takes place on a device with a Wazuh agent installed.
Hence, Wazuh processes filtered alerts coming from Graylog for every endpoint from which logs are collected (including workstations or less sensitive applications, for instance), while directly operating major endpoints (firewalls, domain controllers…) through the Wazuh agent, which allows active responses. Below, the Wazuh dashboard with seven endpoints running the Wazuh or OSSEC agent.
Security and privacy considerations
Ensuring the security and privacy of logs is a fundamental aspect of your log management architecture.
Implementing strict access control measures is essential to protect log data from unauthorized access. Define who can access the log data, make modifications, or delete logs. Use role-based access control (RBAC) to assign specific permissions to users or groups, ensuring that only authorized personnel can interact with log data.
Encrypting log data both in transit and at rest enhances its security. Use secure communication protocols for log transmission and implement encryption for log storage to protect against data interception and tampering. Where log data contains sensitive or personally identifiable information (PII), consider techniques like data anonymization and redaction: this protects privacy while still retaining valuable log data.
Finally, when sending alerts and log data from Graylog to Wazuh, prioritize secure transmission channels, so that data is encrypted and protected in transit to maintain its confidentiality and integrity.
Conclusion
The integration of Graylog and Wazuh offers a powerful solution for log management and cybersecurity. By centralizing log collection, optimizing security, and streamlining threat detection, this architecture empowers organizations to navigate the complex landscape of modern cyber threats. With scalability, data retention, and a strong focus on log security, this approach ensures the long-term health and integrity of digital infrastructures.