As a host-based intrusion detection system (HIDS), OSSEC [1] detects and reacts to security incidents in real time. The software is capable of detecting a wide range of security incidents, including attacks on filesystems and directories, changes to system files and configuration files, failed login attempts, and attempts to escalate privileges. The tool also detects changes to logfiles and network attacks such as port scans, connection breaches, and distributed denial-of-service (DDoS) attacks.

In this article, I show you how to set up the server and the clients. You can also set up OSSEC as a Docker container. All packages are available directly from the download page [2].

Taking Countermeasures

OSSEC offers a range of countermeasures to help you respond to security incidents, such as blocking IP addresses or hosts that exhibit suspicious behavior and terminating processes that are unauthorized or attempting attacks. Additionally, the application lets you disable user accounts that are misused for attacks and supports alerting to ensure a rapid response to incidents. The free software basically focuses on monitoring systems and networks.

In this article, I look at why the use of OSSEC is a sensible step toward significantly enhancing security on networks. Ultimately, OSSEC helps you detect security incidents before virus scanners or other systems, which is particularly important in ransomware attacks, for example, because time is a critical factor.

OSSEC Editions

The basic version of OSSEC is open source and offers you a rich feature set with log-based intrusion detection, rootkit and malware detection, active response, compliance auditing, file integrity monitoring, and system inventory. It is important to note that OSSEC as a HIDS focuses on monitoring individual systems. Therefore, you should use OSSEC in combination with other security tools such as network-based intrusion detection systems (NIDS) or firewalls to create a comprehensive security system.

Other editions are also available. For example, OSSEC+ offers additional functions, such as machine learning, but requires registration with the manufacturer before use. This edition also integrates the Elasticsearch, Logstash, Kibana (ELK) stack. Central administration and thousands of rules, as well as role-based authorizations and a comprehensive reporting system, are restricted to the scope of the commercial-grade Atomic OSSEC version. The differences between the various editions are listed on the download page.

Typical Deployment Scenarios

One practical use of OSSEC is system and application log analysis to detect signs of security breaches or suspicious activity. If the system identifies this kind of activity, OSSEC notifies you by email, Slack, or another configured notification method. At the same time, the software can carry out actions in its active response system, such as blocking users or IP addresses.

File integrity checking is another use case. OSSEC monitors important system files and directories for changes. If something unexpected occurs, you are notified and can actively combat attackers and malware before the damage spreads, making this a valuable tool, especially in the fight against ransomware.

For Windows users, OSSEC also supports monitoring the Windows registry. Changes in the registry can indicate a security breach or an unwanted application. OSSEC tracks these changes and alerts you to suspicious activity.

Rootkit detection is another important feature of OSSEC. Rootkits are malicious programs that try to hide deep in the operating system to remain undetected. The tool searches for known signatures and behavior patterns to identify rootkits and report their presence to you.

Finally, OSSEC has an active response functionality that reacts automatically to detected threats. For example, you can configure OSSEC to block network access for an IP address from which repeated failed login attempts have been made.