April 23, 2010

Schlepping the Firewall Log, Part IV

Posted in Scripting at 8:37 pm by dgcombs

Firewall logs are data. Big data. Lots of data. Add to that logs from abuse messages and failed logins and we’re starting to talk about real disk space. Trying to review all those logs looking for the one that screams, “I found a problem!” is a lot like sifting through the cat’s box. No one wants to do it. And no one wants to hear about what you found. That’s why I thought it’d be better to move all those logs into one place and use some automation to make it easier on me. You can do this with a good bit of money. There are lots of kitty litter sifters around… for a price. Q1Labs makes a very good one. So good, in fact that Juniper OEM’s it as STRM. On the other hand there is Splunk, which markets… Splunk, which has been called Google for Logs. The problem with both of these excellent log sifters is that to get one big enough to handle all the data I’ve got costs a lot of money. So I decided to cobble together a proof of concept using open source software. I hope it works just well enough to convince the powers that be to spend some money, but not well enough that they consider the problem solved.

  • MongoDB: At a core of the system, I used one of the cutting edge, NoSQL databases. It uses JavaScript as a language for core operations and shell.
  • Resin: This open source web server is built on Java and uses JSP scripts for dynamic web pages.
  • FW1-Loggrabber: An Open Source program to pull logs from my Check Point firewalls
  • Python: The open source scripting language to tie it all together.
With all this information together, the remaining mystery is how to display the information in a way that will make sense. While pondering this issue, I hit upon Google’s Chart API. This API makes it straightforward, if not simple, to display a data chart or graph based on information fed to it. The simple Chart API has two flavors, the first one simply shows a static graph. The second one uses a JavaScript library to animate the chart components both to add a little flair and to assist the viewer in understanding what’s being displayed. For simplicity sake, and because the JavaScript library adds to the load time, I opted to use the static graphs to start. This is, after all, a proof of concept.
To use the Google Chart API, it is necessary to craft a URL link that is the source for a graphic element in a web page. For example, the HTML code to show a picture in the web page is
<img src=”picture.png” width=’150′ height=’150′>
This locates an image file located at “picture.png” with a width and height of 150 pixels on the web page. In the Google API, the src= section is replaced by a special URL. Google translates the data in the URL and returns a PNG (Portable Network Graphics) file which the browser embeds into the rendered web page. The URL which shows spam complaints I’ve received over the previous month, looks like this

<img src=”http://chart.apis.google.com/chart?cht=ls





&chtt=Abuse+Complaints+During 4/2010

This is, of course, one long line in the HTML code which is generated by the JSP code discussed below. But it is easier to look at one line at a time. The base portion of the URL is the part that sends your browser to Google’s chart API,
http://chart.apis.google.com/chart to collect the graphics file. The next portion, tacked onto the end, is cht=ls which means this chart type is line. The chs=380×240 defines the chart size. The chart data is defined as type text in the line that starts with chd=t. The chart title is set with the chtt line. The plus signs are replaced by spaces when the chart is rendered by Google. Finally, the chrx line adds range marks on the axes so the lines make some sense. Clearly putting this together manually is cumbersome, but using JSP code embedded in the web page makes it reasonably straightforward.
To begin, the underlying Java engine needs to be able to find the MongoDB Java connector. For this to work, the Mongo JAR file is put in the LIB directory under the ROOT directory.

——+www |

————-+webapps |

————————+ROOT |



Then in the <head> section of the JSP file:

<%@ page import=”java.util.Calendar;”%>

<%@ page import=”com.mongodb.Mongo;”%>

<%@ page import=”com.mongodb.DB;”%>
<%@ page import=”com.mongodb.DBCollection;”%>
<%@ page import=”com.mongodb.BasicDBObject;”%>
<%@ page import=”com.mongodb.DBObject;”%>
<%@ page import=”com.mongodb.DBCursor;”%>

Mongo m = new Mongo();
DB db = m.getDB( “fw” );
DB abuse = m.getDB(“abuses”);
DBCollection coll = db.getCollection(“logs”);
DBCollection abuses = abuse.getCollection(“abuses”);
DBObject myDoc = coll.findOne();

These lines define the Java connector for Mongo. The next set connect to the Mongo Database server. Finally, the connection to the Logs and Abuses collections is defined. It is now possible to use these variables to extract data and use it in the rendered web page.
<!– this is the code for the latest spam complaint listing
The graph covers one month (30 days)
The graph is 380×250 pixels
The graph is labled at the top and along the side
// Get the MONTH and YEAR (as numeric)
Calendar cal = Calendar.getInstance();
int month = cal.get(Calendar.MONTH) + 1;
int year = cal.get(Calendar.YEAR);
String spamurl =”<img src=\”http://chart.apis.google.com/chart“;
spamurl += “?cht=ls”; // Chart Type = Line
spamurl += “&chs=380×250”; // Chart Size = 380×250
spamurl += “&chd=t:”; // Chart Data Introduction
// Find the counts for this month … so far
BasicDBObject query = new BasicDBObject();
DBCursor spams = abuses.find(query);
spamurl += spams.next().get(“count”)+”,”;
spamurl = spamurl.substring(0,spamurl.length()-1);
spamurl += “&chxt=x,y”; // Chart Axes
spamurl += “&chtt=Abuse+Complaints+During ” + month + “/” + year; // Chart Title
spamurl += “&chrx=0,0,30,10|1,0,120,10”; // Chart Axis Labels
spamurl += “\”/>”; // End of Chart URL


The string spamurl is first defined as the core of the URL for the Google Chart API. Then the Chart Type and Size are added. The Query is executed to pull out the information as a MongoDB cursor which is then iterated to extract the count field. Note the line containing the substring call. I found the Google Chart API is very touchy about the format of text data in the URL. Using a loop to extract the data field and attach a comma really throws a wrench into the works. In order to fix this, I used the simplest code I could figure out to remove the final comma. Finally, the next lines added the Axes, labels and ends the URL. The out.println(spamurl) renders the line in the HTML. Note that none of this Java code is sent to the web browser. It is executed on the server and the resulting line is sent to the browser using the out.println command.

Posted via email from Meyeview (Posterous Style)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: