Forwarding log files to Logstash server via SSH

So you have an ELK stack set up and now want to forward your nginx logs to your new shiny cluster. But for some reasons, you are not allowed to install Beats to the machines you are trying to monitor the web server logs (for example shared hosting)

The good thing is, you have access to your log files via SSH.

In that situation, this is what we can do:

in Web Server, rsync the nginx log files to your logstash server
make a cron script that will run every minute that will re-run the rsync

Setup rsync

First, you need to make sure the web server is able to access the logstash server via SSH. You might want to whitelist the web server IP to the logstash server SSH port. Additionally, add the ssh key of web server to the authorized_keys in the logstash server

On Web server

cat ~/.ssh/id_rsa.pub

Copy the output of the above command and login to Logstash Server

vi ~/.ssh/authorized_keys

Paste the ssh key of the web server in a new line, then save the file. Now, in web server, create a shell script called send_logs.sh with the following commands inside:

#!/bin/bash
rsync -avzhe ssh /var/log/nginx/* [email protected]:/home/youruser/logs

Remember to replace the youruser, your.logstashserver.com and /home/youruser/logs with actual values. Those are the logstash server ssh username, hostname of the logstash server and the target folder in the logstash server respectively.

Make sure your make the file executable and do a test run

chmod +x send_logs.sh
./send_logs.sh

Check on the logstash server if the log files was actually sent there (in our example, it is on /home/youruser/logs folder)

Schedule when you want to forward the logs

Once you verified the rsync script is properly working without problem, create a cron entry that will execute this every minute (if course you can change the interval to whenever you like)

crontab -e

then enter in a new line

* * * * * /home/youruser/send_logs.sh

Save the file and exit. It should be now forwarding the logs every minute! The good thing about rsync is it just sends over the recent entries in the log files instead of re-sending everything every-time it will execute. It also uses compression during sending the data so it uses less bandwidth. Additionally, since you are sending stuffs via SSH it is being securely transmitted across the network.
It goes without saying that this setup might incur a delay of 1 minute before you actually see the parsed data into ElasticSearch/Kibana because the log files is only being sent after every minute, but hey, you should be only doing this when it is really impossible for you to install data shippers and agents to the target servers.

Did you find this useful?

I'm always happy to help! You can show your support and appreciation by Buying me a coffee (I love coffee!).