====== NGINX honeypot – the easiest and fastest way to block bots! ======

===== Introduction =====

[[https://www.getpagespeed.com/server-setup/security/nginx-honeypot-the-easiest-and-fastest-way-to-block-bots|Original]]

The Internet is not a safe place these days. Hosting a public website means
exposing it to multiple attacks from evil bots, which, at best will cause extra
CPU and I/O load to your server.

If your web server is NGINX, you may be rightfully tempted to make use of some
3rd party WAF modules to counter the bad guys. One such module is
nginx-module-security, other is NAXSI.

But what if I told you that there’s a trick that would allow your NGINX to
easily filter out 99% of the bots out there, without third-party modules? Read
on to find out how.

===== Know your average enemy (bot) =====

==== Evil bots try to upload. ====

Suppose that you have a WordPress blog, and sure enough, the bad guys are trying
to check if they are able to find a weak spot. They do this by trying different
upload endpoints of various plugins. As an example, one of the bots was trying
to access:

<code text>
/wp-content/plugins/ungallery/source_vuln.php
/wp-content/plugins/barclaycart/uploadify/uploadify.php
/wp-content/plugins/barclaycart/uploadify/settings_auto.php
/wp-content/plugins/hd-webplayer/playlist.php
/wp-content/plugins/cherry-plugin/admin/import-export/upload.php
/wp-content/plugins/viral-optins/api/uploader/file-uploader.php
</code>

Those plugins most likely do not even exist on your website!

So what we can obviously do, is ban any IP that attempts to access a resource
that doesn’t exist on our site. Honeypot resources are either:

Locations that aren’t supposed to be accessed by a genuine user, e.g. not linked
from anywhere, hostnames that are known to be invalid (see below)...

We will now add an NGINX honeypot that will work in a simple and effective way:
when a malicious bot requests a known, yet non-existent upload location, NGINX
will immediately ban their IP.

Let’s dive into implementation details for this kind of ban.

===== Pre-requisites =====

RHEL 7 based system, e.g. CentOS 7, and EPEL repository:

<code bash>
yum -y install epel-release
</code>

==== Setup FirewallD ====

We are going to create two FirewallD IP sets.

The two IP sets’ names are honeypot4 and honeypot6, for IPv4 and IPv6 addresses,
respectively.

Any IP addresses placed on either of the two sets should be banned in the server
firewall. To achieve this, we configure these IP sets to be in FirewallD’s drop
zone.

<code bash>
firewall-cmd --permanent --new-ipset=honeypot4 --type=hash:ip --option=maxelem=1000000 --option=family=inet --option=hashsize=4096
firewall-cmd --permanent --new-ipset=honeypot6 --type=hash:ip --option=maxelem=1000000 --option=family=inet6 --option=hashsize=4096
firewall-cmd --permanent --zone=drop --add-source=ipset:honeypot4 
firewall-cmd --permanent --zone=drop --add-source=ipset:honeypot6 
firewall-cmd --reload
</code>

Next, we’ll teach our NGINX when and how to add IP addresses to those IP sets.

==== Setup NGINX honeypot locations ====

Our “trap” locations in NGINX will forward requests to a special FastCGI daemon
powered by fcgiwrap.

Each of those honeypot locations will include exactly the same directives to
ensure passing requests to block-ip.cgi CGI script, that we’ll create later.

To make things clean, we will include those directives from a separate file:

<code conf>
/etc/nginx/includes/honeypot.conf
    fastcgi_intercept_errors off;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/local/libexec/block-ip.cgi;
    fastcgi_pass unix:/run/fcgiwrap/fcgiwrap-nginx.sock;
    keepalive_timeout 0;
</code>

Add locations for some known plugin endpoints that do not really exist for your
website:

<code conf>
/etc/nginx/sites-available/example.conf
location = /wp-content/plugins/ungallery/source_vuln.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/barclaycart/uploadify/uploadify.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/barclaycart/uploadify/settings_auto.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/hd-webplayer/playlist.php {
   include includes/honeypot.conf;
}

location = /wp-content/plugins/cherry-plugin/admin/import-export/upload.php {
    include includes/honeypot.conf;
}

location = /wp-content/plugins/viral-optins/api/uploader/file-uploader.php {
    include includes/honeypot.conf;
}
</code>

Using exact location matching via = ensures that matching is fast and has
priority over your existing \.php location.

Another approach would be regex matching. It may look somewhat cleaner as you
can simply list all the plugins you don’t have in a single spot:

<code conf>
location ~ ^/wp-content/plugins/(ungallery|barclaycart|viral-optins|hd-webplayer)/ {
    include includes/honeypot.conf;
}

location ~* ^/phpmyadmin/ {
    include global/honeypot.conf;
}
</code>

But hey, is there any good plugin out there that requires direct access to its
PHP files under wp-content? None, of course.

So our honeypot locations can be greatly simplified to block anyone trying to
access PHP files in wp-content.

Furthermore, let’s drop some locations that satisfy the rules “existing
locations that are not meant to be accessed by genuine users”.

Our final configuration snippet becomes:

<code conf>
location ~ ^/wp-content/.*\.php$ {
    include includes/honeypot.conf;
}

location = /wp-config.php { 
    include includes/honeypot.conf;
}

location = /wp-admin/install.php {
    include includes/honeypot.conf;
}

location ~* ^/phpmyadmin/ {
    include global/honeypot.conf;
}
</code>

**TIP:** talking about PhpMyAdmin, don’t be someone who installs it, eh?

The regex matching is prone to configuration errors because the position of
these directives relative to other locations is important. Be sure to place your
honeytrap regexes before your existing \.php location.

==== Install and configure fcgiwrap ====

The fcgiwrap is needed to empower our NGINX with the ability to launch the bash
script to ban IPs in the firewall. Technically, you could use whatever existing
scripting engine currently used for your website (PHP with WordPress). But for
efficiency reasons, let’s avoid launching the PHP interpreter for banning.

<code bash>
yum -y install fcgiwrap
</code>

The fcgiwrap conveniently ships with a service file that allows us to launch
multiple instances using different users. E.g. fcgiwrap@nginx.socket unit will
launch fcgiwrap service with nginx user, and listen on a UNIX socket.

To find the UNIX socket’s path easily, you can run:

<code bash>
systemctl status fcgiwrap@nginx.socket
</code>

This would reveal: /run/fcgiwrap/fcgiwrap-nginx.sock.

Ensure that the service is enabled and running:

<code bash>
systemctl start fcgiwrap@nginx.socket
systemctl enable fcgiwrap@nginx.socket
</code>

==== Create the CGI script ====

Let’s create a CGI script at ''/usr/local/libexec/block-ip.cgi''.

The script will call actual bash script for banning an IP, using sudo:

<code bash>
#!/bin/bash

echo "Status: 410 Gone"
echo "Content-type: text/plain"
echo "Connection: close"
echo

echo "Bye bye, $REMOTE_ADDR!"
sudo /usr/local/bin/block-ip.sh

exit 0
</code>

Any time NGINX will call this script, this will launch sudo
''/usr/local/bin/block-ip.sh'' and pass along REMOTE_ADDR.

Make the CGI script executable:

<code bash>
chmod 0755 /usr/libexec/block-ip.cgi
</code>

==== Note on Keep-Alive ====

NGINX, like any other web server, supports keepalive connections.

Simply blocking an IP in the firewall is not sufficient, because it affects only
future connections.

If bots are smart enough to use Keep-Alive (which is easy to implement), they
can still make malicious requests over the initially established connection.

That’s where keepalive_timeout 0; in our NGINX honeypot include comes in handy.
It instructs NGINX to close the connection with the malicious client, after
blocking CGI script completes.

We also explicitly instruct the malicious client to close the connection via
Connection: close header.

==== Create the bash script to block an IP ====

Let’s create the bash script that will be launched by NGINX via sudo, that will
block an IP address.

<code bash>
#!/bin/bash

if [[ -z ${REMOTE_ADDR} ]]; then
    if [[ -z "$1" ]]; then
        echo "REMOTE_ADDR not set!"
        exit 1
    else
        REMOTE_ADDR=$1
    fi
fi

# Put space separate list of trusted IP addresses, not to lock yourself out 
# if you like to test the honeypot! : )
TRUSTED_IPS=(1.2.3.4 2.3.4.5 127.0.0.1)
if printf '%s\n' ${TRUSTED_IPS[@]} | grep -q -P "^$REMOTE_ADDR\$"; then
    echo "Trusted IP"
    exit 0
fi

if [[ "$REMOTE_ADDR" != "${1#*[0-9].[0-9]}" ]]; then
  /sbin/ipset add honeypot4 ${REMOTE_ADDR}
  /sbin/conntrack -D -s ${REMOTE_ADDR}
elif [[ "$REMOTE_ADDR" != "${1#*:[0-9a-fA-F]}" ]]; then
  /sbin/ipset add honeypot6 ${REMOTE_ADDR}
  /sbin/conntrack -D -s ${REMOTE_ADDR}
else
  echo "Unrecognized IP format '$1'"
fi
</code>

We could also use firewall-cmd to add to IP sets, but of course, we want to
avoid the heavy lifting of the Python interpreter. On a 1 CPU VPS, firewall-cmd
  *-ipset=honeypot4 --add-entry=... runs 0m0.494s while pure binary ipset add
honeypot4 ... took only 0m0.002s to run!

That said, if you don’t care about the millisecond differences between different
ways of blocking, using fds program will yield the cleanest approach.

<code bash>
sudo yum -y install https://extras.getpagespeed.com/release-latest.rpm
sudo yum -y install fds
</code>

With fds, our script becomes:

<code bash>
#!/bin/bash

if [[ -z ${REMOTE_ADDR} ]]; then
    if [[ -z "$1" ]]; then
        echo "REMOTE_ADDR not set!"
        exit 1
    else
        REMOTE_ADDR=$1
    fi
fi

# Put space separate list of trusted IP addresses, not to lock yourself out 
# if you like to test the honeypot! : )
TRUSTED_IPS=(1.2.3.4 2.3.4.5 127.0.0.1)
if printf '%s\n' ${TRUSTED_IPS[@]} | grep -q -P "^$REMOTE_ADDR\$"; then
    echo "Trusted IP"
    exit 0
fi

fds block ${REMOTE_ADDR} --ipset honeypot
sudo!!
</code>

What else is missing? Surely enough NGINX runs under non-privileged user and
can’t sudo /usr/local/bin/block-ip.sh, yet. We want to allow nginx to gain
privileges to run the script as the superuser. Moreover, for security reasons,
we will allow only the REMOTE_ADDR environment variable to be passed while
launching the script.

So create sudoers configuration by running sudo visudo -f
''/etc/sudoers.d/nginx-block-ip''.

This command will open up the system editor (likely vim), in edit mode. Simply
paste in:

<code>
Defaults!/usr/local/bin/block-ip.sh env_keep=REMOTE_ADDR
nginx ALL=(ALL) NOPASSWD: /usr/local/bin/block-ip.sh
</code>

Then close the editor normally, e.g. by typing :wq for vim, or Ctrl+X for nano.

That’s it! Restart NGINX and you’re good to go, NGINX will start banning bots
for you. But there’s, even more, you can do… 🙂

===== Huge Improvement with hostname honeypot =====

Evil bots don’t even know your domain name!

Most of “current wave” bots will only know your IP, because they are scanning
public IPv4 ranges and iterating one IP after another as their victim.

Those bots will share these common characteristics:

Since all they know is your IP, they request resources with either an empty or
fake Host header.

They will only make HTTP requests (not HTTPS)

So you can greatly reduce the load from those bots by blocking any client that
does not provide valid hostnames. Obviously, valid hostnames are simply all
domains that you host on your server, and any other domain would be an invalid
hostname.

In /etc/nginx/nginx.conf, setup a map listing all your website domain names:

<code>
map $http_host $default_host_match {
    getpagespeed.com 1;
    www.getpagespeed.com 1;
    default 0;
}
</code>

In a server location of your websites, add honeypot for bad hostname:

<code>
error_page 410 = @honeypot;
if ($default_host_match = 0) {
    return 410;
}
location @honeypot {
    include includes/honeypot.conf;
}
</code>

With the configuration above, NGINX will check if the requested domain is in the
list specified in nginx.conf. If there is no match, then 410 status code is
returned, which is handled by the @honeypot named location, which, of course,
launches our bash script for banning.

===== Caveats =====

In our important honeypot location ^/wp-content/.*\.php$ which denies access to
all PHP files under /wp-content, there is a slight chance that you have a bad
WordPress plugin that uses just this location to execute its PHP files. Such
plugins should be reported and dealt with. But sure enough you don’t want to
block valid users from your website.

To act out of extra precaution you may want to temporarily return 411; in this
location and monitor your traffic with a script:

<code python>
import os
import re



# grep 411 logs/access.log | grep wp-content > analyze.log

# Open a file for reading
with open('analyze.log', 'r') as f:
    # Read the entire contents of the file
    log = f.read()
    # Print the content
    # extract requested URL
    uris = re.findall(r'"(?:GET|POST) (?P<uri>\S*)', log)
    for uri in uris:
        uri = uri.split('?')[0]
        file_path = "httpdocs" + uri
        print(file_path)
        if os.path.exists(file_path):
            print(file_path + " exists and requested!")
</code>

If the script returns no result, it means there are no actual PHP files exist in
your WordPress plugins which are being accessed. If there are, those plugins
should be removed or replaced.

As a last resort, you should whitelist those files in the honeypot location. For
example if you must have
/wp-content/plugins/wp-invoice/lib/gateways/js/wpi_gateways.js.php executed,
then instead of ^/wp-content/.*\.php$ use
^/wp-content/(?:(?!plugins/wp-invoice/lib/gateways/js/wpi_gateways\.js\.php).)*\.php$.

===== How this compares to anything =====

Surely enough, you should not use this approach alone. There is never “enough
security”, and you should use Fail2ban, Malware Detect, and ModSecurity.

However, we can see how the honeypot approach can complement the mentioned
tools, and soften their disadvantages.

For example:

Fail2ban continuously scans the log files in an attempt for early blocking of
offenders; the honeypot banning matches immediately and reduces both the load
and the log noise tremendously

Malware Detect finds uploaded malware, whereas honeypot banning (and
ModSecurity) prevent their upload in the first place

So other tools may be either slow or “too late”. They are surely enough useful.
But now we can do better with the additional security layer thanks to the NGINX
honeypot approach.