December 8, 2010

Including the Kitchen Async

Posted in Uncategorized at 11:39 am by dgcombs

After completing my little web application to review firewall log data, I noticed something interesting. Most of the time, I got the information back. But on the first attempt of the day, I simply got a blank screen. It was almost as if MongoDB was unable to complete the database query and supply the data before the web browser was finished rendering the page. This was not acceptable. It needed a reply with an answer. Every time.

I inserted some console logging statements in the Node.JS code to see what was going on. It was hilarious to watch NodeJS chugging along finding the answers but having nowhere to send them. Sure enough, the problem was MongoDB slogging through too much data. Even after creating indexes, it appeared to be very slowly working its way through every single record for the first query. But after it had moved the results it needed into memory, the info came back fast enough to be rendered within the web page instead of falling on the floor.

When my feeble attempts to speed up (or “optimize) MongoDB using indexing and refactoring the database, failed, I realized it was time to take a different course of action. Since I had already reduced the number of records I was saving and further consolidated like records using mapReduce, I’d have to look elsewhere for a solution. Fortunately I didn’t have to go far.

To restate the problem, the database query was taking a long time (in compute terms). But the page rendering was relatively quick. This is exactly why NodeJS was written. NodeJS takes advantage of the fact that the program is going to have to wait a long time (again, in compute terms) for a database to query, retrieve, sort and return data. So NodeJS goes off and does something else, like generate and send the rest of the web page, while waiting. The simple solution is to force the async-ness out of NodeJS and just make it sit there and wait for that data to return before it sends the web page to the browser. The interface from NodeJS to MongoDB I’ve been using is a JavaScript module called Mongo-Native. But some intrepid souls from LearnBoost have built “an extremely simple interface for MongoDB” called Mongoose. One of the promises it makes is to “Reduces the burden of dealing with nested callback from async operations.” But to be honest, I could never wrap my head around the models, getters/setters and promises that Mongoose uses. Fortunately, it is built on top of Mongo-Native, so I didn’t tumble down a rabbit hole trying it out. Mongo-Native also allows you to nest async callbacks, effectively making the synchronous. Here’s how the author shows it in the sample code extract:

var db = new Db(‘node-mongo-examples’, new Server(host, port, {}), {native_parser:true});, db) {
    db.collection(‘test’, function(err, collection) {      collection.remove(function(err, collection) {
        collection.insert([{‘a’:1}, {‘a’:2}, {‘b’:3}], function(docs) {
          collection.count(function(err, count) {
            sys.puts(“There are ” + count + ” records.”);
          });        });

I’d seen this kind of thing before in Control Flow in Node and subsequent posts in the series. Like a dim candle cursing the dark, it started to dawn on me that this was not going to help. It doesn’t make sense to have each step wait on the previous step. That is the whole raison d’etre of NodeJS. And besides, forcing a human to wait while the browser did nothing but wait for a web server which is waiting on a database didn’t seem like a good user experience. And anyway, it was simply too much like irony.

I was not surprised that someone had already figured this out years ago and called it AJAX, Asynchronous JavaScript and XML. The fact that I would attempt to solve one async problem using another async technology was just an added bonus. AJAX allows you to pull data from the web server while the browser is rendering the rest of the page. When the data is finally delivered, the browser then displays that information without refreshing the whole page. It sounded like a perfect match. But as the wicked witch said, “These things must be done delicately or you hurt the spell.”

The bucket of water I found for the wicked witch was JQuery, “a fast and concise JavaScript Library that simplifies… Ajax interactions.” Well, that was good enough for me. In fact, JQuery comes with what Rebecca Murphey calls “convenience methods” in her JQuery Fundamentals ebook. Using $.get to request some HTML data seemed to be an easy-as-pie way to accomplish my goal. But how to initiate the request within a request? Delicately, of course!

Express has built-in routing to control the format of the URL and the variables generated in the final request. So I built my routing to use the first letter of the firewall name plus the next three letters to designate the report (service, destination or source) and the final chunk as the date.

app.get(‘/:fw.:rpt?’,function (req, res, next){
  var fw = req.params.fw.substring(0,1).toLowerCase();
  var rpt = req.params.rpt;
  if (fw) {
    // Do Stuff

When I needed to open the appropriate MongoDB collection, I concatenated the components together (Oh, and remember that getMonth gets special treatment):

db.collection(fw+rpt+yesterday.getFullYear()+(yesterday.getMonth()+1)+yesterday.getDate(),function (err, collection){

It works fabulously. Every time. So what did I find people doing?

Firewall Statistics: psvc2010127
svc Count
80 204732
53 151546
135 48261
443 39139
445 24277
1433 16846

Such good users! A little web browsing. A little online shopping. Some file sharing. But wonder if I should be worried about that MS-SQL?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: