Monday, March 30, 2009

New IIS Website Log Cleaner Script

An update to my attempts to create a script to clean up my IIS log files, removing search engine bots and intrusion attacks and just leaving the people visiting websites - I originally used a couple of quite dirty batch files, knowing that, while effective, it was not exactly a well engineeered solution. I've now rewritten it as a Powershell script that will list all the .log files in the directory and remove lines based on keywords - its just a question of getting the keywords right. Controversially I am removing any line with "bot" in, which concievably might remove legitimate traffic.

Incidentally, I used set-content to create the tmp1.txt file and the put data in it because that was the easiest way I found of making sure Powershell didn't create Unicode encoded text files, which my web log statistics program couldn't read.

Anyway, this is the Powershell script:

Get-ChildItem *.log -name > logs.txt
$Logs = Get-Content "logs.txt"
Write-host 'Started Processing...'
ForEach($string in $Logs )
Write-host 'Processing...' $string
copy-item $string backup
$null Set-Content tmp1.txt
cat $string where { $_ -notlike "*basicstate*" -and $_ -notlike "*slurp*" -and $_ -
notlike "*Ask+Jeeves*" -and $_ -notlike "*bot*" -and $_ -notlike "*DECLARE*" -and $_ -
notlike "*blog-preview*" -and $_ -notlike "*HostTracker*" } set-content tmp1.txt
remove-item $string
ren tmp1.txt $string }

Friday, March 27, 2009

Speeding up Aero on Windows 7 and Vista

Running Windows 7 Beta on a Netbook has made me look a bit at how to make Aero a bit faster and whether it was worth having it on at all. Second question first - your PC should be faster with Aero ON. Vista especially on my Netbook was painfully slow without Aero - the reason being the main CPU and memory are doing a lot more graphics work with Aero off - turning it on shifts a whole heap of work to the Graphics processor and memory.

That doesn't mean we need every bell and whistle though, and I've found its best to remove some of Aero's effects.

To get to this page, right click Computer, click Properties, Advanced System Settings and the top Settings button, under performance. Select "adjust for best performance", then put a couple of things back on. I would suggest ticking the boxes in this picture (taken from Windows 7 Beta) - "Enable Desktop Composition" and "Use visual styles on windows and buttons" are essential as disabling these would turn off Aero. The rest are personal preference, the ones I have on here make little difference to the speed of Windows but make things more readable and useful.

Thursday, March 26, 2009

Outlook 2007 chronically slow on Server 2008

Just spend a very frustrating hour and a half installing updates and hitting Google repeatedly trying to fix an apparently simple problem - Outlook 2007 SP1 installed on Windows Server 2008 x64 being slow. And I don't mean "a bit sluggish", I mean, click on the icon and literally make a cup of tea before it opens. Anyway, tried every update released (nothing), Safe Mode (still nothing), streaming it with Xenapp 5 or installing it locally (same problem) and swearing (made me feel better).

Eventually I tried a command I read on a forum post about a similar problem and it worked straight away!

Run CMD as Administrator (right click command prompt, Run As Administrator and typed
"netsh interface tcp set global autotuninglevel=disable"

And rebooted. All sorted, flies along now. Well, I say flies along, it moves at a usable speed - so normal then. Going to do this on all our Xenapp 5 Windows 2008 servers now since they all seem to have the same issue.

Here's a post about what this changes:

Saturday, March 21, 2009

Asus EEE 901 - fixing the black screen problem

Since I got it a couple of says ago me and my new Netbook have been on something of a journey - it is on its forth operating system already, if you count Linux, which barely made it out of the packaging before it annoyed me. This journey would have been much happier and smoother if it was not for one very common problem - generally known on the forums as the "Blank Screen" or "Black Screen" bug.

With XP and Vista, install was as expected until I installed the correct driver from the Intel site for the 945 Express graphics chip. After the first reboot the usual windows loading screens happen then all goes black. From the sounds and drive activity it seems to be still running but is not usable. Windows 7 had the same issue but this had the driver already, so it just went black as soon as installation finished.

A quick fix is to plug in a monitor and press fn-F5 to switch to it, then switch back, when the screen carries on working fine. Crazy!

Some digging on the forums later and it is a bug with every BIOS update for the EEE 901 after 1101, which was quite an early release. So until it is fixed properly, this is the procedure to fix the problem:

1. Get a USB stick that you don't mind formatting
2. Download and install the HP USB format tool (you don't need any HP gear to use this!)
3. Get some DOS system files - if you don't have any use these:
4. Run the tool - choose your USB drive, select Quick Format and Create a DOS Startup Disk, and navigate to the files you just got ready.
5. Download the v1101 bios from the ASUS site
6. Rename your 1101.rom file to 901.rom and copy it to your now bootable USB stick
7. With the USB stick in your EEE 901 boot it up and press ALT-F2 when its doing its BIOS checks - it should now do lots of updating. Let this process finish and it should reboot.
8. It will tell you to look at the BIOS settings, this is a good idea so press F1 when promtped. Load default settings, adjust as wanted and save and exit.

You should now be able to actually use a graphics driver!

Thursday, March 19, 2009

Dual booting into Windows 7 an Asus EEE 901

I recently managed to convince myself that the thing my life was lacking was a new toy - since I have a baby and a toddler toys are in no short supply in my house, but the generally crinkle or squeak rather than run beta operating systems so I decided to jump on the netbook bandwagon. After lots of reading reviews I decided Asus were still where it was at, and went for an Asus eee 901 Linux.

Linux lasted about half an hour, and that wasn't just because I was unable to persuade it to use my wireless network. I bought the netbook at least partly because I wanted to play about with installing Windows 7 beta on it - at least partly just for the challenge.

Thing is, I can't see what the fuss is about. After reading tutorials like this I thought this was going to be rocket science, in fact it seemed so complicated, with bootable SD cards (don't have one) and messing about with vlite and the Windows AIK and using Startup Repair from the DVD that I decided to just just sick a DVD in the (external) drive and see what happened. Actually, I installed XP first (while Windows 7 build 7057 downloaded) on the 4gb partition, then formatted the 15gb partition and installed Windows 7 to there. Job done - I now am dual booting Windows 7 and XP, so I can decide which one is faster (obviously XP) and better (obviously Windows 7).

Anyway, since everyone else is publishing guides to getting Windows 7 on a Netbook, this is mine for dual booting XP and Windows 7. If somoeone can tell me why mine seem to have half the steps of everyone else's that'd be great! Oh, and this works just fine with Vista, I found 7 to be much faster than Vista on the 901 though.

1. Make sure you are running bios version 1001 or 1101 - these are the only current versions that will work in XP and Windows 7 without an external monitor!
2. Plug in your external DVD drive and change the bios to boot off it.
3. Pop in your Windows XP Professional CD and boot
4. Delete all partitions on the 4gb and 16gb drives. Create partitions on both and install to the 4gb one.
5. Get the latest drivers from the Asus site, install everything and connect to the (wireless?) lan.
6. Format the 16gb D: drive and copy the contents of the Windows 7 DVD to it. Run Setup
7. Install Windows 7 to the big partition.
8. After install, remove the Windows 7 installation files, they're just taking up space.

Yep, that's it. Anyway, I will soon make a decision about which OS to go for (probably 7) and rebuild to save space, but for now its all working lovely.

Just got to get used to tiny keyboard now.

Tuesday, March 3, 2009

De-googling my website logs

One of the reasons I have a website is fascination with the logs that are generated by IIS, seeing what browser and operating system visitors use, how many each day, what they type into Google and Live Search (and other search engines of course, but to be fair two thirds of my traffic is from Google and Live is almost everything else). Yes, you are being watched!

I use WebLog Expert to analyse the logs every few days which makes lovely HTML reports with graphs and stuff. For instance, here are the operating systems you're all using!

I've always thought though that the impressive list of visitors each day were being inflated by search engine bots, sites I use to check the site is up by "visiting" it and frankly attempts to hack the site. There was a spike in traffic in August 2008 for instance that turned out to be a 3 day long SQL injection attack! Of course I never knew how many were real people.

Anyway, I decided to make them "pure", just real people, no automated traffic. I discarded logs from before 2008 but had over a year of log files - about 450 text files, most of them hundreds of lines, so I clearly needed something to process them. IIS 7 logfiles are quite simple - each line is a visitor doing something - downloading an image for instance - with lots of information on the same line, like the previous site visited, IP address, OS, etc. You coming to this page will have generated quite a few lines of text. The good thing though is that if you can find the lines to remove all the information is very cleanly removed.

So, I started looking at what to remove from the logs. Most of the rubbish lines were from search engine bots which I started to identify one at a time - Googlebot, MSNbot, etc - until the penny dropped they all had bot in their name. I found the SQL injection attacks all start DECLARE. I excluded basicstate, hosttracker and blogger, since any line with those in are basically me. In the end, the list of strings to look for to identify unwanted lines are (so far):
- bot
- ysearch/slurp
- basicstate
- HostTracker

I will find a better way to do this soon, but the first attempt is:
1. Make a text file of all the log file names (Open CMD, "dir *.log /b > list.txt")
2. Rename list.txt to a batch file, add "call processlog " before each log file name
3. Create a batch file called processlog.bat to do the procecssing.

The batch file is below - basically it reads the whole contents of the file minus any lines with a string in to a temporary file, then to another temporary file against another string, until its done, then replaces the original file. Simple! Okay, this is a very inefficient method but it worked just fine. It incidentally reduced my stats by about 75%!!! But I know they are real now....
copy %1 backup /Y

type %1 find /v "bot" > tmp1.txt

type tmp1.txt find /v "basicstate" > tmp2.txt

type tmp2.txt find /v "ysearch/slurp" > tmp1.txt

type tmp1.txt find /v "DECLARE" > tmp2.txt

type tmp2.txt find /v "" > tmp1.txt

type tmp1.txt find /v "HostTracker" > tmp2.txt

del %1 /Q

del tmp1.txt

ren tmp2.txt %1