April 9, 2010

Schlepping the Firewall Log, Part III

Posted in Uncategorized at 5:30 pm by dgcombs

Using the firewall log data already gathered, we were able to have some impact on the number of bots and bot-like activity. In the Windows world, the program nbtstat is very useful in finding out what machine owns a specific IP address. This is handy if you are running Windows. However, the system I am using for collecting and examining my log files is running Ubuntu Linux Server edition. I thought perhaps that nbtstat was not going to be an option. However, back in June 2003, Jim Halfpenny ported a C program based on nbtstat to PERL. Now I could quickly feed an IP addresses to nbtstat.pl and it would cough up a machine name! I handed the resulting information to the IT department manager who quickly identified at least two of his own people! Confident that we're on the right track, I forged ahead.

Having a few firewall logs that span some time available for serious in depth analysis makes all the difference. At least I know that if I see some activity relating to bots or viruses I can raise an an alert. So I'm automating the process. The goals for this week are pretty straightforward:

  • Pull all the firewall logs from yesterday to the server
  • Make sure the time/date field is set to UNIX
  • Recalculate the IP Address fields to a long integer
  • Import the file into a MongoDB database
Some time ago, the program fw1-loggrabber was written by folks wanting to access their Check Point log files. The source code was place on Sourceforge.net. The most recent version, 1.11.1, was posted in 2005. However, it still works quite well. It uses the LEA protocol which is part of an overall suite that Check Point calls Open Platform for Security. LEA stands for "Log Export API." Exactly what we need for this project. Although five years old, this same fw1-loggrabber has been modified and used in several other systems including the commercial product Splunk and a SANS certification project by Mark Stingley.

As I mentioned, the system is running Ubuntu Server 64-bit. This is by design. The MongoDB 32-bit version does not permit any databases greater than 2Gb. Since I wanted to keep more than 2Gb of data on the server, I opted for the 64-bit version. This caused the fw1-loggrabber binary to stop working with a cryptic message:

file not found 

But of course it found it. It just couldn't run the 32-bit libraries on which fw1-loggrabber depended. Looking inside the file, I saw that it was looking for a 32-bit version of libc6.so. So I consulted the Ubuntu packages website and found that I could remedy this problem by installing a pair of 32-bit libraries.

sudo apt-get libc6-i386

sudo apt-get ia32-libs

Now fw1-loggrabber ran fine. Using the instructions on the Splunk website, I configured a secure connection to the Check Point firewall log server.

One thing I wanted to do was modify the time/date field which Check Point by default generates as something like 3Feb2004 14:15:16. This would clearly be difficult to search, select and index. However, one of the options in the fw1-loggrabber configuration file allows you to get standard UNIX time. This is defined as the number of seconds elapsed since January 1, 1970. This very large number is convenient to store as an unsigned long integer and allows for convenient indexing. Insofar as humans are concerned, it's not terribly readable. However, it's easy to convert into something humans understand.

The IP address fields are another matter entirely. These are usually represented as a four numbers separated by dots: "". As a string, it's easy to read for humans. However, for computers, it must be treated as a string of characters. This does not lead to simple indexing or searching. MySQL, C and even Python have built-in functions called ATON and NTOA which do convert from this string notation to a "packed" integer. Unfortunately, the packed integer does not fit well in a MongoDB. So I opted to use the Python code below to create my own ATON function:




                for quad in ip.split('.'):

                        intip=intip+(int(quad)*(256 ** exp))





The first part allows me to quickly loop through a firewall log line. Anything that doesn't resemble an IP address is simply returned. Anything that does is converted into a very large number.

A default MongoDB installation comes with an import file that accepts information from STDIN making it a perfect recipient for the lines coming from fw1-loggrabber and the Python interpreter.

/usr/local/fw1-loggrabber/bin/fw1-loggrabber –logfile <yesterday>  -l /usr/local/fw1-loggrabber/etc/lea.conf -c /usr/local/fw1-loggrabber/etc/fw1-loggrabber.conf |

/usr/local/fw1-loggrabber/bin/import_log.py |

/usr/local/mongo/bin/mongoimport -h localhost -d fw -c logs –type csv –headerline

By executing the whole thing at 1am using cron, I can check first thing in the morning for any nefarious activity during the previous day and dig down with ad hoc queries against the newly imported data. 

Posted via email from Meyeview (Posterous Style)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: