.NET to the rescue

I’ve got a request from a customer to scan or review the firewall logs because of an audit finding.  But there is no budget to purchase any management product to actively monitor the humongous logs coming in daily.  We’ll have to make do with what we’ve got.  Well, we do have a state-of-the-art PC, armed with Intel Core 2 Duo @ 2.33GHz and 2GB of RAM.  Cool!

A typical daily firewall log size is around 3.8GB (roughly 11.8 million lines of text entries).  At first, we tried using some freeware tools out in the web like “Splunk”.  But because of the size of the log files, the response is slow and also it doesn’t meet our requirements.  We wanted something like a “Top 3 Deny entries”.

Well… being MS centric guy, I know about this great tool (if not the greatest tool) ever released free by Microsoft.  I’ve used it before and the speed is simply FAST and the beauty of it all is it’s flexible enough to scan any type of log files.

This is what I’ve done:

1. Run logparser to extract the key fields like “Action, Source IP, Destination IP, Source Port, Destination Port” and output to a csv file
2. Run logparser again to churn out the Top 3 Deny entries from the output csv file from Step 1

And guess what?  For this 3.8GB (11.8 million lines) text file, it only took 3.5 minutes!  Pretty impressive yeah?

Observations

While running, I noticed via “Windows Task Manager” that logparser only uses maximum 50% of the CPU.  Most likely it’s due to the fact that logparser (version 2.2) has been around for quite some time and it’s not optimized to make full use of multi-threading capability to use up all the dual core CPU power.  I do hope someone in MS can release a newer version to harness the power and imagine that instead of 3.5 mins, it will be 1.75 mins!!! ;)

Code Examples

Ok, as requested by Chris, here is the code that does the magic…

A typical Juniper FW log looks something like this:

Nov 5 23:58:11 192.168.1.3 Netscreen-FW1: NetScreen device_id=Netscreen-FW1 [Root]system-notification-00257(traffic): start_time="2008-11-05 23:56:32" duration=0 policy_id=125 service=syslog proto=17 src zone=Untrust dst zone=Trust action=Deny sent=0 rcvd=0 src=172.26.1.75 dst=166.2.3.50 src_port=514 dst_port=514 session_id=0

Use this command to extract the important parameters from the log:

Private Function ScanLog(strFile)
	Dim strSQL

	strSQL = "SELECT EXTRACT_VALUE(Text,'action',' ') AS Action, " & _
		"EXTRACT_VALUE(Text,'src',' ') AS Src, " & _
		"EXTRACT_VALUE(Text,'dst',' ') AS Dst, " & _
		"EXTRACT_VALUE(Text,'src_port',' ') AS Src_Port, " & _
		"EXTRACT_VALUE(Text,'dst_port',' ') AS Dst_Port " & _
		"from " & strFile & " to results.csv"
		
	WshShell.Run LOGPARSER & " -i:TEXTLINE """ & strSQL & """", HIDE_WINDOW, WAIT_ON_RETURN
End Function

The above will output to a csv textfile call “results.csv”.  Next, use this function to generate the Top 10 results.

Private Function GenTopResult(strFile)
	Dim strSQL

	strSQL = "SELECT TOP 10 Action, Src, Dst,Src_Port, Dst_Port, COUNT(*) AS Hits FROM " & strFile & " to top.csv WHERE Action='Deny' GROUP BY Action, Src, Dst, Src_Port, Dst_Port ORDER BY Hits DESC"
			 
	WshShell.Run LOGPARSER & " -i:CSV """ & strSQL & """", HIDE_WINDOW, WAIT_ON_RETURN
End Function

At the end of it, you will get a “top.csv” text file containing the Top 10 results sorted by highest hits.

1. Install the Technical Preview of Live Writer
2. Install the official WLW onto a Windows XP/Vista machine
3. Copy & overwrite the WLW folder from the XP/Vista machine to Server 2008 (C:\Program Files (x86)\Windows Live\Writer)

I wanted a syntax or code highlighter solution for my site (BlogEngine) but have no lady luck in getting it up on the first 3 attempts and spent quite a good amount of time troubleshooting :(

The Syntax Highlighter solution by Chris Blankenship that I feel is the best so far is:

• http://www.dscoduc.com/post/2008/05/01/SyntaxHighlighter-Extension-for-BlogEngineNET.aspx 
• http://www.dscoduc.com/post/2008/09/25/More-Efficient-SyntaxHighlighter.aspx 

Well, the mistake I’ve made is that in order for this to work, you MUST use Windows Live Writer with the plugin to post.  This is the caveat.

$j = jQuery.noConflict();	
$j(document).ready(function() {

    var shBasePath = 'dp.SyntaxHighlighter/';
    var shScriptPath = '/js.axd?path=' + shBasePath + 'Scripts/';
    
    $j.getScript(shScriptPath + 'shCore.js', function(){
        dp.SyntaxHighlighter.ClipboardSwf = shScriptPath + 'clipboard.swf';
        $j('head').append('<link href="/css.axd?name=' + shBasePath + 'Styles/SyntaxHighlighter.css" rel="stylesheet" type="text/css" />');
        ...  
    };  
})

Yes!!! It’s finally working! :)  Thanks to Chris.  I didn’t know that I must use it with Windows Live Writer.

PS: I’m now using the jQuery method.  The cool thing is that it helps to save your bandwidth by downloading only the necessary scripts.

After I have signed up with GoDaddy on the Windows Hosting plan, I've found out that there are actually alot of complaints and grumbles on the many limitations and poor technical support on GoDaddy.

Well, things that are cheap are of course comes at a price. Yes, GoDaddy's windows shared hosting is cheap, but there are workarounds.... so not to worry. Let's get started on the tips & tricks.

Tips & Tricks

Tip 1: SQL Express

If you're using windows shared hosting, you don't have access to the server to run the aspnet_regsql.exe against it. GoDaddy is very strict about what remote connections it allows to the live database. Also, be aware that GoDaddy doesn't support SQL Express (like those mdf database you normally saved it in /Apps_Data folder) - you'll be using its big brother SQL Server. But that's not a problem as the basic plan comes with 1 free SQL DB! (How cool is that?!)

Basically, follow these steps:

1. Create a new database using the GoDaddy control panel. Because you are sharing the database server with other databases, you'll only be allowed use a name that doesn't already exist. This should explain why you can't call it ASPNETDB - someone else has probably snapped that one up.

2. If you need to install the ASP.Net Security Schema - there's an option on the "Create Database" GoDaddy configuration page. Check the option when you're creating the database and all the tables and stored procedures will be installed in the database.

That's about it. Good luck.

Tip 2: Email settings

To setup email on BlogEngine.NET on Godaddy, configure the email settings as follows:

The SMTP relay server is relay-hosting.secureserver.net

• IIS prompting for credentials event though you are logged into the domain and have configured IE to pass credentials automatically.
• Not being able to log into the site no matter what username/password you use
• Getting HTTP 404.1 when trying to access the site
• Authentication events in the security event log like Event ID 537 : An error occurred during logon

If you have encountered these problems, your issues may be related to a new feature added to Windows Server 2003 SP1 and Windows XP SP2. The feature at fault is the Loopback check security feature that helps mitigate reflection attacks that was added to the operating system in these service pack releases. A detailed description of the issue can be found at:   http://support.microsoft.com/?kbid=896861

1. open regedit
2. Find HKLM\System\CurrentControlSet\Control\Lsa
3. Create a new DWORD value called DisableLoopbackCheck and give it a value of 1
4. Restart the computer

1. Open regedit
2. Find HKLM\system\CurrentControlSet\Control\Lsa\MSV1_0
3. Create a new multi-string value called BackConnectionHostNames
4. Add the host headers to this value that map back to the local loopback
5. Restart your computer

For my case, I've tried Method 1 on a Windows Server 2003 SP2 and it didnt' work. However, Method 2 works on my Windows Server 2008 (and it doesn't require a reboot)

BlogEngine on GoDaddy

Hi, welcome to my blog. In case you are curious on what engine this blog is running on, it's BlogEngine.NET 1.4.5.  No surprises here since I specializes in MS technologies :)

You can find out more about it at http://www.dotnetblogengine.net/

Theme

The current theme I'm using right now is Dilectio.  I downloaded it from http://designdisease.com/portfolio/dilectio_wordpress_theme/ on 02/11/2008.  At that time, there are a coupole of bugs which you may want to go around it.

Issue 1: It will cause a horizontal scroll bar on IE for 1024 x 768 resolution

You just need to edit the following to solve it:

wp-content\themes\dilectio\style.css:

#bgcontainer - width: 999px
.Header - width: 999px
.CON - width: 969px
.Footer - width: 969px

Issue 2: Text appears on the right side of the image

Edit style.css and remove the following:

/**/ * html .PostContent p img {} /* */
.PostContent p img { max-width: 420px;}
.PostContent p img { width:expression(this.width > 420 ? “420px” : this.width); }
.PostContent img { margin: 0px 10px 10px 0px; float: left; border:solid 1px #495b6b;}

Hosting

After searching Google, GoDaddy.com is by far the only web hosting company that offers everything (ASP.NET, SQL, MySQL) at a bargain price! They even give 10GB of space + 30GB of web traffic.  What's more, $1.99 for domain name.  What more can you ask for?

Be sure to check out the promo codes that can save you another $5 ;)

Tips & Tricks to get it up & running on GoDaddy

1. Upload the entire BlogEngine files to your web site at GoDaddy using any FTP software

2. Open up GoDaddy and the Site Configuration manager

3. Go to File Manager

4. Check Mark the App_Data folder and then click the permissions button

5. Set the access to Read, Write and Reset all children to inherit.  This will allow all files under it to inherit the above settings.  This allows you to move files in and out of App_Data folder and not having to keep updating the permissions in GoDaddy.

6. That's it.  Happy blogging!