Stefkoff's Blog

In these days, cyber security is one of the important topic that everyone should consider. It goes from the local router up to the production environments and Cloud infrastructure.

Currently, there are a lot of open-source tools for testing vulnerabilities of an app, that they can be used even for gaining access over it. Also, on the other hand, a lot of users are using open-sourced applications, like nginx, apache, etc, that the attacker could find the exact version of the application and find how it can be broken or even gain access over some private resources.

There are a lot of Vulnerabilities databases, like OVS, that the hacker could just know the version of the application, searches in the OVS for the particular version and do some mess with you server.

That it why, you should always needs to update your software to the latest versions, because the developers are giving more and more security fixes for the application.

My infrastructure

Current, I have a bunch of Dell servers in my basement, that there are VMWare vSphere Hypervisor with a lot of virtual machines that I'm using for my public or private apps. This blog is server from one of the virtual machines in my basement.

For managing the internet access, I'm using 2 Mikrotik and two ISPs. Mikrotiks are responsible for the NAT, port forwarding and Firewall.

I'm also managing some servers from my clients, since I work as a DevOps. There are all in the cloud.

What is the problem

The general problem with all open servers (server with opened port, like 22 SSH), is that there are a lot of bots that are searching for a IP addresses and opened ports. Port 22 is mostly used, because it can grant access to the entire system.

Since I have a GitLab instance in one of my virtual servers, my port 22 is also opened. If I check the Auth log: tail -f /var/log/auth.log I can see that almost every minute (even less) there is someone that is trying to access throw ssh via some random username and password. And this is not only in my server. This is valid for all of the servers across the World, that have some port opened.

There is another BIG issue that I have - in my routes, I have a VPN configuration and since the VPN is using IPSEC for authentication, my ports for IPSEC are opened also. When I open the logs from the Mikrotik, I can see the same situation - a lot of attackers are trying to negotiate with my IPSEC policies by guessing username and passwords. If this happens, it will be very bad - Mikrotik will assign an IP address of the attacker and he will be part of my local network and will have access over all of the assets.

Any solutions?

Yea, there are a lot of solutions for this. One of the best solutions, but may be written as *"workaround** is to allow the access for you applications (ssh, nginx, apache, etc) only by the local network, et Listen 192.168.0.100 an example from Apache configuration file, that will listen on the private assigned address from the router, and can be accesses only from the local network (of no port forwarding is made). This is good, but if you need to access your apps from remote location, you have to open a VPN connection and you can end up with some opened ports and the issue that I was explaining above with IPSEC.

What I'm using:

I'm using a series for tools (applications) that helps me to block most of the attackers. Here is the list of tools that I'm using and later on I will explain what each of this is doing:

• Graylog - used for Syslog messaging and alert notification
• fail2ban - used to catch the unauthorized access some application
• python application - written by my, simple application that receives an IP address and add it to the routes blocked list
• rsyslog - UNIX application that manages syslog and remote syslog servers

With combination of all of these tools, the following situation can happen:

• If an attacker is trying to connect to IPSEC with an invalid credentials, Mikrotik will send a syslog message to the graylog server. Once the syslog message is received in the Graylog server, it is process though an Alerts with a specific pattern, check for IPSEC negotiations fails and it will send an HTTP request to the python application
• If fail2ban catches someone to try to access with invalid credentials, then an request is send to the python application with the given IP address and iptables role is added to the server (default behavior for fail2ban) to block the feature access for the given IP address)
• if the python application receives an IP address, then it checks if it some local or reserved address and if not, it is adding the address to the Mikrotik's Blocked List. This blocked list is used to filter the IP addresses that are trying to enter inside the router (inside my entire network).

In my case, for three months, I end up with more than 4500 addresses that are blocked in my Mikrotik. And the list grows every day.

So here is a breakdown for each of the tools that I'm using and their pseudo configuration

Rsyslog

Rsyslog is a UNIX tools that can read from the syslog (/var/log/syslog) and send the logs to a remote server. You can read more about the syslog in Wikipedia. Rsyslog can be configured, that once a Critical error is received in the syslog, by some application, then it will be send over a remote location with opened UDP 514 port (default syslog port or any other configured). In my case, in all of my servers (local or remote), every syslog notification, except for INFO, are send to my graylog server.

Graylog server

The Graylog server is configured to receive an Syslog UDP packets. The configuration is simple as it is.

Graylog, have a really good Alert manager, that can be used to analyze the messages, and execute some actions, like - send an email, executing HTTP request and so on. I use the alerts for if I receive some authentication error message, the it will be posted to the python application for analyzing and blocking the IP address

fail2ban

fail2ban is maybe, one of the most used open-source application for analyzing logs and blocking IP address. I use it in all of my servers, that have public access, It most power is to analyze the auth log and find what IP address is trying to reach your system with unauthorized access. The default behavior in fail2ban is to add an iptables record to block the given IP address access to some resource or to entire system (all ports).

fail2ban can be very customizable and it have a lot of built-in filters and actions that can be using. Filter is basically the functionality that analyze the log for for a specific application log, finds if there is some unauthorized access and do something with the IP or the domain.

In my case, I'm using two actions - one to ban the IP in the server (throw iptables) and the other one is the send a HTTP request with the given IP address to the python application, so the URL could be added to the Mikrotik's firewall.

fail2ban can be configured in a lot of more complex way, and it can be used for varius situations and not only for ssh

python application

If you are waiting to show you some source code - I will not, because this blog post will become unreadable. Here is a breakdown of what this app is doing:

1. Using Flask the application exposes an TCP port that everyone could send HTTP request.
2. There is only one route that is protected via static Basic Authentication.
3. Once an request is received and authorized, it search for an IP address in the body parameters
4. If an IP address if founds, it compares it for local address and some reserved on.
5. If it not part of the reserved address or not part of the local network, then it is send to the Mikrotik's opened API to add the IP address to a Firewall address list, called "Blocked address"

If you need the source code of the application, you can send be an email to georgi@stefkoff.com

Conclusion

As you can see, the setup is not one of the complex one, but in this way, can can guarantee that these attackers, once they try to break into my system, they will not have another chance.

Also, this configurations is implemented in all of the remote servers, so once some hacker is trying to break throw one of the production servers, it will be added in my local Firewall and I can prevent a feature attacks from him/her.

So, to sumarize - if you are working as developer or DevOps, you have to be sure that you are trying to limit the attackers as more as possible. If you do not do this, later you may end up to getting help from Cyber Security Services and this can cost you a lot of money.

And remember: Your system will NEVER be secured. There will always come a guy that finds a way to break it - even if it is an simple user that do some stupid things and brings DoS by mistake.

If you need more information of my implementation, you can write in the commends below or mail me at georgi@stefkoff.com.

Have a secured day! :)