Monitoring Node Security Using OSSEC Server and Slack
Guide to monitor your node security with OSSEC and Slack.
🤖 Prerequisites
Ubuntu Server or Ubuntu Desktop installed
SSH server installed
a SSH client or terminal window access
🔓 Recommended Stake Pool Security
If you need ideas on how to harden your stake pool's nodes, refer to
Hardening an Ubuntu ServerThese are very recommended steps to perform before configuring a monitoring service.
📐 Install packages
Some packages will be required for the installation. You will need
libz
libssl
libpcre2
libevent
make, gcc
On Ubuntu run this command to install them
🛸 Download OSSEC
First of all, we will need to get a fresh copy of OSSEC. We will download it from github and then checkout the latest stable version.
Go to this page and find out the latest version number.
Then, login to your server and
🛠️ Install OSSEC
OSSEC can be installed in server, agent, local or hybrid mode. This installation is for monitoring the server that OSSEC is installed on. That means a local installation.
You will be guided in the installation by a bash script. As you will see, at this stage we will keep the default settings for most of the installation. To initiate it, type
The installation script will prompt you with some questions. We start with installation language
Here we will use this language so press ENTER.
Press ENTER to start the installation process
Type local for local installation
ENTER for default location. For safety reasons, this location will be accessible only to root user. You don't want anyone to read and potentially find ways to exploit information about your OSSEC configuration.
Type n and ENTER. We will have slack notifications so there's no need to turn on mail notifications. If you want it anyway, you will need to provide an email address with a valid SMTP server.
Press ENTER to confirm. You will want to have an integrity check daemon to checksum important files and monitor any change.
Press ENTER to confirm. Rootkits are malicious software designed to allow illicit access to protected parts of the system. This detection is typically achieved looking for virus "signatures", by integrity checking and by monitoring system usage.
This will be a key step for our Slack setup. We want active response to get immediately notified of all threats. Press ENTER.
OSSEC can provide a firewall drop rule for local firewall in response to high level threats. Optionally, you can whitelist the addresses that you use to connect to the server.
Configuration is done. The script reminds you that some log files will be monitored by default. We will add specific folders and files later. Press ENTER to start to compile and install OSSEC.
Congratulations! You just completed the installation.
✨ Create a new service for OSSEC autostart
Now we want OSSEC to run automatically at startup. To achieve this, we will create a systemd service.
For now we won't start the service, as we want to configure the integration with Slack notifications.
🔮 Configure your Slack workspace
Go to your Slack workspace and create a private channel for each of the server that you want to monitor. You really don't want your monitoring information to be public. Let's say we call our channel ossec-ubuntu as the name of the server.
We will first create a Slack App for the OSSEC service. Go to this page
https://api.slack.com/apps?new_app=1
and click Create New App. Enter OSSEC as a name and select your workspace. Next click on the OSSEC app to view the API information.
Next click on Add features and functionality and Incoming Webhooks. Here we will add an new webhook for the server. Click on Add New Webhook to Workspace and select the channel you want to post to.
Then copy the WebHook URL from webhooks page.
The information we need will be as follows
We are ready to configure OSSEC with our Slack information.
👾 Let OSSEC talk to your Slack API
OSSEC uses a bash script to forward its notifications to Slack. We will first configure this
and modify the empty fields SLACKUSER, CHANNEL and SITE with the information you prepared in the previous session. Eventually, the first part of the script will look like this
Next, we will edit the main ossec config file to actually forward the notifications to this script
and add these lines at the end of the commands section
Current level of notifications for this command is 3. This means that you will be notified for example of all succeeded events, such as ssh logins or sudo commands. This ensures a very good level of protection, but if you think that these notifications are not useful, you can adjust the level. I would suggest raising this to the bearable minimum. I actually keep this at 3 to get notifications for all logins to my servers. As you will see, you will also fine-tune the notifications to your system in the last section.
OK, now let's start OSSEC and see if everything works. Type
If everything is ok, you should get a notification of level 3 with comment OSSEC started in your slack channel.
We will stop it again for now, as we have to make a few more changes.
🍰 Stake pool specific configuration for OSSEC
OSSEC has the capacity to monitor realtime changes to folder. To use this feature, we will edit the ossec.conf file. Type
Look for the <directories> entries and modify as below
Here we added the keywords report_changes="yes" realtime="yes to all directories to monitor their realtime changes. We also want to monitor specific files in the cardano node folder. We will then add a line like this one
where /home/cardano/pool is our cardano node folder. Change this directory to point to your node directory. It's also a good idea to exclude the database and logs folder, so we will add also the following lines to the <ignore> fields.
Another important point is that we want to spot any potential DoS attack to our relay nodes. To achieve this we will monitor the number of connections on the node port from each server and track when a server has more than 2 connections. To monitor the number of connections on a specific port we will use this command
where you will change <YOUR-PORT> with the node port (say 3001 or 6000 for example).
The output will be something like this
In this case, we want to track all IPs that have more than 2 connections at the same time. To achieve this we will add an awk '$1 > 2 { print $2}' at the end. So our final tracker will be this one, to be inserted at the end of the <localfile> commands section. Remember to change <YOUR-PORT> with the actual node port.
Save and close this file. Finally, we want a tracer for errors in topologyUpdater log. We can log its output directly to syslog by adding this line at the top of the script.
Save and close this file. We will also add a decoder to let OSSEC know the correct interpretation of this file. Open the decoder.xml file
and add these lines at the end
If you are using Systemd scripts for your cardano node and have a SyslogIdentifier=cardano-node line in it, we can also add this decoder line to identify the node logs
And later we will see rules to filter by severity. Save and close this file.
Now let's check that everything works. Run this command
and paste this test line then press ENTER. The line should be recognized and decoded, such as in this example
The result show be like this one
Here you see that OSSEC correctly identified the decoder to use and separated the fields in the right way.
Good! We defined all trackers and files to monitor. Save and close this file. Next step will be the definition of local rules.
🔎 Setting rules and fine-tuning
OSSEC comes with a number of predefined rules that implement the best practices in server security. However, sometimes they are a bit too invasive or need to be adapted to the specific server you are running. Each set of rules comes in a .xml file. In this section, we will configure the local_rules.xml file to adapt them to our cardano node.
To edit your rules type
First of all, we will create a new rule to silence all warnings coming from loop devices having no space. Rules have a unique ID (in this case we will use consecutive numbers starting from 100,001) and a level of security. Level 0 means no notification. This rule will be based on rule 531, which is activated in case a device has no empty space, and will be give level 0, i.e. silence, all string matching the regular expression defined in the pcre2 fields (perl regular expression format). We also ad a human format description.
Next, we will configure a rule with high severity to keep track of the incoming connections with more than 2 connections. Don't forget to change <YOUR-PORT> with the actual node port.
We also want to track errors on topologyUpdater so we will add these lines at the end after the </group> line. Here we are creating a new group for our topologyUpdate decoder, which by default has an alert level of 0 and an alert level of 4 for invalid block errors (usually when the node is not working) and level of 3 (low level notification) when topologyUpdater performs more than one request per hour.
Finally, we will add a last group for the cardano node logs
Save and close this file and restart OSSEC. If you let it run for an hour or so, you'll see that you will get a lot of notifications to your OSSEC channel. For example, you may want to adjust the level of the cardano-node rules above, even shutting down some of them by setting the level to 0.
Congrats on completing the guide. ✨
This guide was contributed by Billionaire Pool. If you find this guide useful, please consider staking to it (BIL ticker). Thank you 🙏
Join @BillionairePool on Twitter or hang out and chat at on Discord @
The guide is kindly hosted by our Coincashew friends. Use cointr.ee to find our donation addresses. 🙏
Hang out and chat with our stake pool community on Telegram @ https://t.me/coincashew
🚀 References
Last updated