Blog Posts

Early in 2015 one of the accounts on our server got hacked and was used to send out truckloads of spam emails. It took several hours for me to become aware of the issue, unfortunately. I only realized it once the number of bounces escalated to the point that our server was being blocked by other servers.

The account password was changed and some discussions were held with the user to see what the source of the unauthorized access may have been. The only thing that stood out was the user had been in several cafes with an iPhone the last couple of days, otherwise it was a pretty normal week with usage primarily at work and home. The password was a reasonable length, meaningless and random enough. My guess is the account name and password had been picked up by some sort of Wi-Fi scanner at a cafe a couple days prior to the attack beginning.

In the hours that the attack was in full force, the server’s IP address began to get banned for sending spam, which is how the issue came to my attention. That resulted in me going to several sites to indicate the problem was cleared and I was taking steps to prevent it from recurring.

The first thing I did was turn off any cleartext access to the accounts. The server has an SSL certificate, but a couple of the email accounts, including the compromised one, had cleartext as one of the acceptable connection methods. I also needed to get a lot smarter about the Adaptive Firewall (AF) included with Mac OS X Server. I have a few ebooks on Mac OS X Server that cover AF in some detail, and there are several websites, as well as Apple Discussion boards that cover the topic.

I realized AF was not running correctly. By correctly, I mean it was running, but not looking for anything. I also learned that AF’s default setting is to block an IP address for 15 minutes at a time, then allow the IP to resume accessing the server. I thought that was pretty weak, so I based the blocking time on the number of times the IP address was trying to access the server in a given time. In other words, if attacker 1 had tried 20 times in one hour, they would get a shorter timeout than attacker 2 trying 200 times in one hour. Also, the number of attacks would accumulate over time, so if attacker 1 eventually reached 200 times in 1 day or 60 days, they would eventually end up with the same timeout as attacker 2. Actually more than attacker 2, since they would have been blocked and unblocked when the timeout expired several times over the given time period.

The implementation for this required a database and a program to access the log files where the attacks were being recorded. The database has about the simplest structure you can get, comprising of a column for the IP address, and one for the DateTime of the attack attempt by the IP address. Here’s the MySQL structure statement:

CREATE TABLE `AuthFail` (
    `IP` int(10) unsigned NOT NULL DEFAULT '0',
    `DT` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
    PRIMARY KEY (`IP`,`DT`),
    KEY `idx_DT` (`DT`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8

Note that IP is an unsigned int, not a (15 character) string. Though I’m barely aware of it on a regular basis, the “dotted quad” IP addresses are just a human-readable version of an integer, hence an unsigned int for the IP address. And DT is a datetime, of course. The primary key contains both the IP and DT columns. An additional key for DT is added for queries on the datetime alone. MySQL contains the INET_ATON and INET_NTOA translation functions, which are used to handle the IP addresses in either the dotted quad or integer formats, as needed.

In the next part of this series, I’ll set up the log file parsing program.