It's a common sight and source of annoyance for many system administrators: people attempting to do brute force hacking on your server. Often the /var/log/auth.log file will contain a truckload of error messages and authentication failures.
A common error message would be the "reverse mapping" message:
Feb 15 17:41:58 ams01 sshd[1516]: reverse mapping checking getaddrinfo for user.145.126.222.zhong-ren.net [222.126.145.202] failed - POSSIBLE BREAK-IN ATTEMPT!Another variant is that the reverse DNS lookup doesn't match the forward DNS:
Feb 19 00:55:28 ams01 sshd[32031]: Address 212.156.126.210 maps to 212.156.126.210.static.turktelekom.com.tr, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!Or the log file contains a long chain of error messages where someone is trying to log on with a single user, using a vast array of passwords:
Feb 22 12:01:24 ams01 sshd[77812]: error: PAM: authentication error for root from 60.54.248.46 Feb 22 12:01:27 ams01 sshd[77815]: error: PAM: authentication error for root from 60.54.248.46 Feb 22 12:01:31 ams01 sshd[77818]: error: PAM: authentication error for root from 60.54.248.46And in another scenario, the log file could contain a single IP that is trying to log on with a variety of usernames and passwords:
Feb 23 13:53:01 ams01 sshd[473]: Invalid user db2inst1 from 222.174.35.3 Feb 23 13:53:10 ams01 sshd[477]: Invalid user prueba from 222.174.35.3 Feb 23 13:53:19 ams01 sshd[481]: Invalid user postgres from 222.174.35.3
There are many variants of brute force hacking, and they can be annoying as hell; especially if your server sends you a daily security report containing all the log file entries. I don't know about you, but seeing all those hacking attempts makes me nervous... what if one of them succeeds ? Given enough time, one of them has to come up with my username/password combination... I don't like that idea at all! Fortunately, there are things like bruteblock. Bruteblock is a little program that does just that: it blocks brute force hacking attempts (Yay! ).
The way it works is pretty simple: it reads the log entry, checks it against some regular expressions and if it registers X amount of matches within Y seconds, it adds a table to the firewall to block that IP address. Of course this could lead to some problems, because some ISP's have dynamic IP address allocations, so you could be blocking half a country after a while. Bruteblock solves this by adding a timestamp to the firewall table. A small daemon that runs in the background monitors the firewall table, and after an X amount of time removes the blockade.
The first step is to tell your logging facility that it needs to pipe the authentication messages to bruteblock. This is easily done by adding a single line to /etc/syslog.conf:
auth.info;authpriv.info |exec /usr/local/sbin/bruteblock -f /usr/local/etc/bruteblock/ssh.confWe don't want to restart the syslog daemon just yet, because we still have to determine what bruteblock should check and how it should handle matches against the regular expressions. /usr/local/etc/bruteblock/ssh.conf contains some pre-fabricated regular expressions for common sshd authentication failures. To get rid of even more log file spam and brute force attempts we will add an additional two regular expressions:
regexp4 = sshd.*reverse mapping checking getaddrinfo for \S+ \[(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\] failed - POSSIBLE BREAK-IN ATTEMPT!
regexp5 = sshd.*Address (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) maps to \S+, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
The configuration file also contains some settings on how many matches per how many seconds it should detect before blocking an IP address, a duration for the blockade and which firewall table to use for the IP addresses. The default duration is 600 seconds (10 minutes), which we will change to 84600 seconds (24 hours). The other settings can be left at to: 4 matches in 60 seconds will trigger action, and the IP addresses will be added to table 1.
Next, we'll want to decide what to do with the IP addresses that are trying to do brute-force hacking on us. We'll add a few rules to /etc/rc.firewall:
## Block SSH brute force scanners.
##
${fwcmd} add deny all from me to "table(1)"
${fwcmd} add deny all from "table(1)" to me
We'll want to place these rules as early as possible in the chain of rules for performance issues; I'll write a separate article on rule-chains sometime in the (near) future. Next we can either restart the entire firewall, or just add the two rules manually by adding a chain number; just be sure to add the rules to the firewall script as well, so they don't get lost at the next reboot.
Allright, after we've added the rules to the firewall, we can restart syslogd and start bruteblockd (the daemon that handles the cleanup for the firewall table):
/etc/rc.d/syslogd restart /usr/local/etc/rc.d/bruteblockd start
And that's pretty much all there is to it. After a few days we can take a peek at /var/log/auth.log and see something like:
Feb 28 07:39:21 ams01 sshd[96155]: Did not receive identification string from 220.172.191.31 Feb 28 07:43:24 ams01 sshd[96253]: Invalid user arun from 220.172.191.31 Feb 28 07:43:27 ams01 sshd[96257]: Invalid user arun from 220.172.191.31 Feb 28 07:43:31 ams01 sshd[96261]: Invalid user arun from 220.172.191.31 Feb 28 07:43:34 ams01 sshd[96265]: Invalid user arun from 220.172.191.31 Feb 28 07:43:34 ams01 bruteblock[95217]: Adding 220.172.191.31 to the ipfw table 1Great, It looks like it works! Bruteblock matches 4 rules, and at the 5th it adds the IP address to the firewall table. A quick peek in at the firewall table shows that the IP address has been added, including a timestamp:
ams01# ipfw table 1 list 124.95.128.162/32 1330485366 193.34.145.55/32 1330527853 200.113.185.227/32 1330491753 213.165.165.130/32 1330487447 220.172.191.31/32 1330496014And the firewall rules also confirm that traffic has been blocked:
ams01# ipfw -a list | grep table 01200 210 103040 deny ip from me to table(1) 01300 152 14488 deny ip from table(1) to me
So with a few simple steps, brute force hackers will have to be extremely patient and switch IP addresses every 4 attempts. It won't stop hackers completely, but it will definitely make it a bit harder for them
As most other system administrators, I put a lot of value in having a stable server. Unfortunately it is always possible that, for whatever reason, your server "hangs" and becomes unresponsive. One of the most common reasons is a Denial of Service attack (and sometimes bugged anti-virus software ) which generates 100% CPU usage and causes your server to become unresponsive.
To prevent stuff like this from happening, something called a watchdog was invented. The basic principle is real simple: the watchdog has to be reset within X seconds, or else the system will reboot. FreeBSD has support for both hard- and software based watchdogs. Since my server has an Intel ICHxx chipset, I logically opted for the hardware based solution.
Before making permanent changes to my kernel, with the possibility of wrecking my server, I had to determine if my server would actually support the interface. Since my server has an elevated kernel security level I first had to reboot it with level 0 security before being able to load kernel modules:
ams01# kldload ichwdNothing happened, the world did not implode on itself, my server did not suddenly reboot itself; This was a good sign. Fetching a list of the loaded kernel modules confirmed that the module was in fact loaded:
ams01# kldstat Id Refs Address Size Name 1 7 0xffffffff80100000 6abc20 kernel 2 1 0xffffffff807ac000 8b8 accf_data.ko 3 1 0xffffffff807ad000 1580 accf_http.ko 4 1 0xffffffff807af000 3818 ichwd.ko
And consequently, a quick peek in the kernel boot messages also told me that the interface was recognized and support:
ams01# dmesg | grep ichwd ichwd module loaded ichwd0: on isa0 ichwd0: Intel ICH9R watchdog timer (ICH9 or equivalent)
Excellent! Of course loading a kernel module manually would mean that it would not be loaded anymore after the first reboot, and I still had to reboot the server to restore the kernel security level). I had two options now: either I compile a new kernel with the ichwd device enabled, or I tell the system to load up the kernel module at boot-time. I decided to go for the second option:
ams01# echo 'ichwd_load="YES"' >> /boot/loader.conf
Once I update the system to a newer release of FreeBSD, I have to compile a new kernel anyway, but for now this will do just fine. The next step was to enable the watchdog daemon that will be doing the polling:
ams01# echo 'watchdogd_enable="YES"' >> /etc/rc.conf ams01# /etc/rc.d/watchdogd start
I let the server run for a few minutes and nothing happened; which is good... it should only do something if something is wrong, after all. Since I had to reboot the server anyway to restore the kernel security level, and I wanted to see what would happen if something did go wrong, I killed the watchdogd process and waited. A few seconds later, suddenly my SSH connection was terminated. About 30 seconds later I received a text message on my phone that the server had rebooted itself.
Well well... It seems to work just fine! I sincerely hope that I never actually have to use this failsafe though
Back in 2008, I bought a Western Digital MyBook "World" 500GB NAS. It was a decent NAS, but mine had some cooling issues which were probably related to where I kept it. It was on top of a closet, where it was pretty dusty and warm, which isn't very beneficial for hard disks. So after a while it started developing issues: it would randomly give time-outs and become unresponsive. After a reboot it would work for a while, but after a week or so it would give timeouts again.
I quickly moved all essential data to my computer's hard disk, and powered down the NAS until I had found a replacement and would copy the rest of the data as well. But I made a cardinal sin when I powered down the NAS: I made an assumption. As we all know, assumption is the mother of all fuckups and Finagle's law will apply. I assumed that by turning off the NAS via the big button on the front, it would completely shut down the NAS. Unfortunately, I found out that this was not the case when I had a new storage device a while back and was ready to migrate all my data. The NAS felt quite warm, which surprised me to say the least. Apparently when you push the big "on/off" button at the front, it only powers down the little main board; but it keeps the hard disks spinning. To power down the hard disks, you have to unplug the power supply or flip a small switch on the back of the device. When I tried to boot up the NAS, it didn't do anything at all.
Crap! now what ? Did I just lose a large portion of my data ? I wasn't ready to give up just yet. My geek-credibility would be at stake. Since the NAS was essentially dead, I carefully dismantled it and took the Hard drive out. Because i didn't know what the state of the hard drive would be, I didn't want to plug it directly onto my computer's main board; after all, Finagle's law was in effect and I didn't want to blow up my main board just yet. I went online and bought a SATA/IDE to USB 2.0 Adapter. Since it was a Linux based NAS, I installed Ubuntu Linux on a USB stick and booted up my laptop.
First order of business was to determine if the Hard drive would be salvage-able. As soon as I plugged in the power to the hard drive I could hear a soft whirl indicating that it was still spinning up. Since it would be the third SATA device (the first being my laptop's internal hard drive, and the DVD-ROM the second) connected to my laptop, it would have to be /dev/sdc that I was looking for. A quick peek in the boot log confirmed that the system had indeed detected the hard drive. Next up was to see if the hard drive would still operate, and which part I had to restore:
root@ubuntu:/# sfdisk -l /dev/sdc Disk /dev/sdc: 60801 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdc1 3 368 366 2939895 fd Linux raid autodetect /dev/sdc2 369 381 13 104422+ fd Linux raid autodetect /dev/sdc3 382 504 123 987997+ fd Linux raid autodetect /dev/sdc4 505 60800 60296 484327620 fd Linux raid autodetect
Well, well... this is interesting. Even though the NAS only had one hard drive, Western Digital still made all the partitions part of a raid set. My best guess is that Western Digital did that because the MyBook line also featured larger models with two hard drives in it; and they would be able to use the same configuration for those. In my opinion a sloppy solution, but more about that in a moment. In any case, the device showed four partitions, of which one was roughly 480GB. This was clearly my target. Since you can't just mount a raid partition, I first had to find a way to recover the raid set:
root@ubuntu:/# mdadm --examine --scan /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4 ARRAY /dev/md4 UUID=dee2de1f:eec30950:726acf0d:54e86bff ARRAY /dev/md3 UUID=e1e3ad6e:0753b0ae:86339bf0:ca6e9d49 ARRAY /dev/md2 UUID=51a4640a:c27e3a2e:4b454248:eeac819a ARRAY /dev/md1 UUID=6afd8edb:48c319d6:c6d492e3:868da876
Good news so far, Linux recognized the arrays. Now we could attempt to reassemble the raid sets:
root@ubuntu:~# mdadm --assemble --scan mdadm: /dev/md1 has been started with 1 drive (out of 2). mdadm: /dev/md3 has been started with 1 drive (out of 2). mdadm: /dev/md4 has been started with 1 drive (out of 2).
Of course, with a raid1 array the system would expect two drives. But since the NAS only had one hard drive, it would mean that the array would always be in a "degraded" state. This won't stop the array from working, but to me it seems rather sloppy. It also seemed strange to me that it wouldn't reassemble /dev/md2, but since /dev/md4 was my main concern and the hard drive would be scrapped anyway, I couldn't be bothered to investigate it further. Fortunately, /dev/md4 was reassembled properly and I was able to mount it and access my files:
root@ubuntu:~# mount /dev/md4 /media/nas_backup root@ubuntu:~# cd /media/nas_backup root@ubuntu:/media/nas_backup# ls -l total 24 drwxr-xr-x 2 root www-data 4096 2009-04-25 18:10 backup drwx------ 2 root root 16384 2002-02-28 12:30 lost+found drwxr-xr-x 3 www-data www-data 4096 2010-05-07 05:31 PUBLIC
Copying the data took quite a bit longer than I thought. It was only 93GB, but it took well over 3 hours to copy it from the hard drive to my new storage system. A possible cause for the lack of speed was the fact that it was an USB adapter, and I'm not sure if I actually hooked it up to a USB 2.0 port on my laptop. On the other hand, the hard drive wasn't in the best state either, so that could also have been a bottleneck. It was totally worth the wait though; my data was secured.
*phew*
Microsoft Windows... the world's most used-, and probably the most flamed at, Operating System in the world. It's packed with many nifty features, funky eye-candy and unfortunately also many, many bugs.
A quick search on Google told me that this happens when some Dynamic Link Libraries are no longer registered with Windows XP. What the hell? It's a fresh install, I only ran Windows Update, and the system is broken... nice going Microsoft! Fortunately, the site also told how it can be fixed, so without further a due...
When your Windows Update suddenly stops working without any errorcode, just open a command prompt, and enter the following commands:
regsvr32 wuapi.dll regsvr32 wuaueng.dll regsvr32 wuaueng1.dll regsvr32 wucltui.dll regsvr32 wups.dll regsvr32 wups2.dll regsvr32 wuweb.dll
Then re-run Windows Update, and it should work again
Database replication basically means that you keep a continuously synchronized mirror of a database on two or more servers. If one of the servers changes something in the database, it is immediately synchronized to the other server(s). This has a couple of benefits. The most common use is load balancing; spreading the database over multiple servers greatly enhances the performance of large databases and databases with high traffic (e.g. real busy websites). This also brings the benefit of redundancy. If one of the database servers suddenly kicks the bucket, you still have the data on the other server(s).
All in all, it has a couple of benefits that, in certain situations, could outweigh the costs of additional servers.
For this example I will assume you have at least some basic knowledge of databases in general and know your way around a FreeBSD, UNIX or Linux system. Of course, the database replication will work just fine on a Windows based server as well, but obviously the paths and daemon control commands will be a bit different.
I will show you how to set up replication for a single database in a master-master situation. This means both servers are master for- and slave to- each other.
For this example I will assume the following information:
- The database name is exampledb.
- Server_1 has IP address 192.168.1.1.
- Server_2 has IP address 192.168.1.2.
First, we need to add a user that has replication rights. Log on to MySQL with a user that has GRANT privileges (or just use root), and issue the following commands on the MySQL prompt:
GRANT REPLICATION SLAVE ON *.exampledb TO replication@192.168.1.2 IDENTIFIED BY 'password'; FLUSH PRIVILEGES;
Obviously you could replace the *.exampledb by *.*, so that the user can replicate all databases. You could also replace the replication@192.168.1.2 by replication@%, so that the user can log on from any IP address (as long as the firewall permits ). Personally, I prefer to keep things tight and secure, so I hardcode the databases and IP addresses.
Next up, we need to collect some information from the database. On the MySQL prompt, issue the following commands:
USE exampledb; FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS;
The SHOW MASTER STATUS; command is the important one. It should give us an output that is some what like this:
+---------------+----------+--------------+------------------+ | File | Position | Binlog_do_db | Binlog_ignore_db | +---------------+----------+--------------+------------------+ | mysql-bin.001 | 112 | exampledb | | +---------------+----------+--------------+------------------+ 1 row in set (0.00 sec)
We need to write down the values for File and Position. We need these values later on when we configure Server_2.
After that, we make some changes to the MySQL configuration file, which is usually found in the database directory. Default installations of MySQL usually put this file at either /var/db/mysql/my.cnf, some Linux distributions put it at /etc/mysql/my.cnf and if you've installed MySQL via the FreeBSD ports, it will be /usr/local/etc/my.cnf. Add the following information at the [mysqld] section:
server-id = 1 replicate-same-server-id = 0 auto-increment-increment = 2 auto-increment-offset = 1 master-host = 192.168.1.2 master-user = replication master-password = password master-connect-retry = 60 replicate-do-db = exampledb log-bin = /var/log/mysql/mysql-bin.log binlog-do-db = exampledb
When that is done, we restart MySQL. On FreeBSD the proper command would be /usr/local/etc/rc.d/mysql-server restart, and on most Linux distributions it would be /etc/init.d/mysql restart.
Now, we need to make sure that Server_2 has the initial database, so replication can take place. For this we simply make a dump of the database on Server_1, and copy the file over to Server_2, where we import it again. The simplest way to make a dump of the database is the following commands:
mysqldump -u root -p exampledb ] exampledb.sql tar zfvc exampledb.tgz exampledb.sql
Lastly, we need to unlock the tables, so they can be replicated. Log back on to MySQL and enter the following commands on the MySQL prompt:
UNLOCK TABLES; quit;
Right.. onward to Server_2!
First of all, we need to create the initial database, by importing the database dump file that we created on Server_1. This is easily done using the following commands:
tar zfvx exampledb.tgz mysql -u root p < exampledb.sql
Configuration-wise, Server_2 is practically identical to Server_1, with exception of the IP address. So, we log on to MySQL and create a new user, and show the information we need:
GRANT REPLICATION SLAVE ON *.exampledb TO replication@192.168.1.1 IDENTIFIED BY 'password'; FLUSH PRIVILEGES; USE exampledb; FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS;
As before, we write down the values for File and Position, and move on the MySQL configuration file. Add the following information at the [mysqld] section:
server-id = 1 replicate-same-server-id = 0 auto-increment-increment = 2 auto-increment-offset = 1 master-host = 192.168.1.1 master-user = replication master-password = password master-connect-retry = 60 replicate-do-db = exampledb log-bin = /var/log/mysql/mysql-bin.log binlog-do-db = exampledb
When that is done, we restart MySQL.
After MySQL is restarted, we initiate the actual replication. In this case, Server_1 is the master, and Server_2 is the slave. On the MySQL prompt, enter the following commands:
UNLOCK TABLES; LOAD DATA FROM MASTER; SLAVE STOP; CHANGE MASTER TO MASTER_HOST='192.168.1.1', MASTER_USER='replication, MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.001', MASTER_LOG_POS=112; START SLAVE; quit;
If we get no error messages, the replication should work. We can check this with the SHOW SLAVE STATUS\G command. Please note how I use \G to end the command, as opposed to the regular ; (semilicon). The \G command forces the results to appear vertically instead of horizontally. Since the output gives quite a lot of information, having the results appear vertically makes it a lot easier to read. The command should give an output somewhat like this:
*************************** 1. row ***************************
Master_Host: 192.168.1.1
Master_User: replication
Master_Port: 3306
Connect_retry: 60
Master_Log_File: mysql-bin.001
Read_Master_Log_Pos: 112
Relay_Log_File: mysql-bin.006
Relay_Log_Pos: 175
Relay_Master_Log_File: mysql-bin.001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_do_db: exampledb
Replicate_ignore_db:
Last_errno: 0
Last_error:
Skip_counter: 0
Exec_master_log_pos: 112
Relay_log_space: 175
Yes, that is a lot of information, but the most important ones to us are Slave_IO_Running and Slave_SQL_Running. If both of these are running then everything is fine, and we can move on to the last step.
So Server_2 is now replicating properly with Server_1, but since its a master-master situation, we want the replication to occur in both directions. For this to happen, we need to initiate the replication on Server_1, so that Server_1 is the slave, and Server_2 is the master. On the MySQL prompt at Server_1, enter the following commands:
LOAD DATA FROM MASTER; SLAVE STOP; CHANGE MASTER TO MASTER_HOST='192.168.1.2', MASTER_USER='replication', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.006', MASTER_LOG_POS=175; START SLAVE; quit;
Again we can check the status of the replication with the SHOW SLAVE STATUS\G command. If both Slave_IO_Running and Slave_SQL_Running are running then were done!
The databases should now synchronize automatically. Add some records on both servers and see if they show up on the other as well.
Note that on older versions of MySQL (lower than 4.1.1), there are some limitations:
- The USER(), UUID(), LOAD_FILE(), and CONNECTION_ID() functions do not work reliably on a slave (they are replicated without changes).
- The FLUSH, ANALYZE, OPTIMIZE, and REPAIR statements are not replicated. This means that if you change permissions on the one of the servers by editing the tables directly, you will need to FLUSH PRIVILEGES manually on the other server(s).