Ajira.eu

As most other system administrators, I put a lot of value in having a stable server. Unfortunately it is always possible that, for whatever reason, your server "hangs" and becomes unresponsive. One of the most common reasons is a Denial of Service attack (and sometimes bugged anti-virus software ) which generates 100% CPU usage and causes your server to become unresponsive.

To prevent stuff like this from happening, something called a watchdog was invented. The basic principle is real simple: the watchdog has to be reset within X seconds, or else the system will reboot. FreeBSD has support for both hard- and software based watchdogs. Since my server has an Intel ICHxx chipset, I logically opted for the hardware based solution.

Before making permanent changes to my kernel, with the possibility of wrecking my server, I had to determine if my server would actually support the interface. Since my server has an elevated kernel security level I first had to reboot it with level 0 security before being able to load kernel modules:

ams01# kldload ichwd

Nothing happened, the world did not implode on itself, my server did not suddenly reboot itself; This was a good sign. Fetching a list of the loaded kernel modules confirmed that the module was in fact loaded:

ams01# kldstat
Id Refs Address            Size     Name
 1    7 0xffffffff80100000 6abc20   kernel
 2    1 0xffffffff807ac000 8b8      accf_data.ko
 3    1 0xffffffff807ad000 1580     accf_http.ko
 4    1 0xffffffff807af000 3818     ichwd.ko

And consequently, a quick peek in dmesg also told me that the interface was recognized and support:

ichwd0:  on isa0
ichwd0: Intel ICH9R watchdog timer (ICH9 or equivalent)

Excellent! Of course loading a kernel module manually would mean that it would not be loaded anymore after the first reboot, and I still had to reboot the server to restore the kernel security level). I had two options now: either I compile a new kernel with the "ichwd" device enabled, or I tell the system to load up the kernel module at boot-time. I decided to go for the second option:

echo 'ichwd_load="YES"' >> /boot/loader.conf

Once I update the system to a newer release of FreeBSD, I have to compile a new kernel anyway, but for now this will do just fine. The next step was to enable the watchdog daemon that will be doing the polling:

echo 'watchdogd_enable="YES"' >> /etc/rc.conf
/etc/rc.d/watchdogd start

I let the server run for a few minutes and nothing happened; which is good... it should only do something if something is wrong, after all. Since I had to reboot the server anyway to restore the kernel security level, and I wanted to see what would happen if something did go wrong, I killed the watchdogd process and waited. A few seconds later, suddenly my SSH connection was terminated. About 30 seconds later I received a text message on my phone that the server had rebooted itself.

Well well... It seems to work just fine! I sincerely hope that I never actually have to use this failsafe though

It took some blood, sweat, tears and a lot of gasoline; but we're back on the air, and we're cruising on FreeBSD

After postponing, delaying and deferring the issue for quite a bit of time; it was getting kind of embarrassing to put off the migration, and the worst part was that I didn't have an excuse not to do it. I had picked a date in my agenda to do the actual migration; which was on a Friday. But on Thursday I was bored, and decided to do it one day earlier. That decision may or may not have been rushed by the fact that my server was having yet another issue with the virus scanning software.

I downloaded FreeBSD-8.2-RELEASE-amd64-disc1.iso, made a final backup of my server data and got ready to make my way to the datacenter where the server is hosted. You can enter the datacenter 24/7, but they do require you to register on a website so they know who is coming. While trying to register I got an error on the website. I emailed the hosting company that I was unable to register on the website, but that I was en-route and would need access to the datacenter.

When I got to the datacenter and tried to log in, the system said there was no registration for me and therefore it could not let me in. I called the hosting company's helpdesk to ask why they hadn't arranged for access. The guy on the phone said that they had fixed the problem that was preventing me from registering, and that I should be able to register now. I told him that I was already at the datacenter, and asked if he could register access for me. He told me that they're not allowed to do that, and suggested that I use my smart phone to register. I told him that I had already tried that, but the website didn't work because it redirected to some kind of status page as soon as it detected that I was using a smart phone instead of a desktop pc. After some arguing with the helpdesk about how I would get access to the server without having to drive back to my home or harass Daniel at work, the security guard of the datacenter offered me use of his private laptop to register for access. Some bro-fists were exchanged and I was finally able to go inside.

I hooked up my USB CD-ROM player to the server, and made it boot from CD... or so I thought! While trying to boot, it got stuck halfway in loading the kernel. Switching USB ports, rubbing the CD; none of it seemed to help. Man, I was pissed! But I also facepalmed, because I neglected to check if the CD was working before driving off to the datacenter. I bro-fisted the security guard again, told him I would be back in a bit, and drove back home grumpy and hungry.Back home I downloaded FreeBSD-8.2-RELEASE-amd64-bootonly.iso to save some time. I double and triple checked that the CD was booting and working properly. A quick bite later I was on my way back to the datacenter. I hooked up the CD-ROM player to the server again and... it got stuck halfway in loading the kernel again! Needless to say, a small mushroom cloud would have manifested itself above the datacenter. I looked around the datacenter to see if someone else was there. I got lucky; some American guy was working on a couple of servers and had a CD-ROM with him that I was able to lend for a few minutes. Unfortunately, it gave the same result as with my own CD-ROM.

After cooling down a bit, I decided to bring the server home to figure out what the deal was. the brand of CD-R's, a driver issue, a BIOS configuration issue, the ISO's being broken... It could be a lot of different things. Back at home I decided to download FreeBSD-8.2-RELEASE-amd64-memstick.img and try to boot from an USB memory stick instead; which worked perfect the first time; man I was relieved! Since it was already late I decided to continue the next morning.

The next day, everything went as planned. I installed FreeBSD on the server, did some minimal configuration so that I would at least be able to receive some email, compiled a custom kernel and drove back to the datacenter to shove the server back in the rack. The rest of the weekend I spent tweaking the configuration and debugging some PHP scripts to fix case sensitive pathnames, etc..

Over the next few days or weeks I will probably need to do some minor tweaks, but right now I have everything running pretty much the way I wanted, and couldn't be more happy with it. It's so nice when everything works out the way you had it in mind.

During the "downtime" caused by the Kaspersky update, I started to browse for alternatives. One of the alternatives that caught my eye was Clamav, an open-source virus scanner for UNIX systems; Although there is also a Windows port available. As I was peeking a bit through the options and features, an idea sparked in my mind; A memory of an old love that popped up, so to speak.

I tried to dismiss the idea but it kept haunting me, and eventually I surrendered to the unspoken desire: I wanted my old love back, no matter what it takes.

In the last week of January 2011, version 8.2 of the FreeBSD operating system will be released. You might wonder why I'm mention this on November 2nd, but it has a reason. Basically I've got 3 months to freshen up my UNIX skills, convert my sites and services so that they can work with FreeBSD and work out some new stuff. I've installed version 8.1 on my laptop, which will serve as a staging / development template.

I've added a link in the menu to give an overview of the project status. I've done a lot of research and all the issues that made me decide to migrate to Windows in January 2009 are no longer an issue. Maybe I was just lazy back then, or maybe I was just tired of doing the research... Whatever the real reasons were, they're off the past. My love for the FreeBSD operating system is revitalized and stronger than ever. After 2 years of Windows, we're going back to FreeBSD!

Apparently something is wrong with the latest update from Kaspersky Anti-virus, because the last few days the CPU load on my server has skyrocketed to 80-100% load on average. This is caused by two worker processes from Kaspersky Anti-virus (kavfswp.exe) that take up 40-50% each. I've never had this problem before, and reinstalling the software temporarily fixes it, but as soon as it kicks in an update cycle for the anti-virus definitions, it starts all over again.

I'm not too happy with my server having high load. Aside from slowing down my websites, it also consumes more power and I don't know how happy the datacenter is with that. Technically I'm allowed to use 400mA for the server, but due to this nice CPU load bug It's been pumping 464mA. Some searching on Google only told me that in 2009 there was a similar problem. It was caused by an error in the anti-virus definitions and it was solved a week later when Kaspersky released new anti-virus definitions. I hope it's a similar issue, and that it will be fixed soon.

I could disable the anti-virus for the time being, but I don't know if that's such a good idea. Sure, I'm the only one that uploads files to it; but still... I don't like the idea of using an unprotected server. I've temporarily disabled videos till the problem is solved.

Update October 25th, 2010 - 12:19
It seems that I'm not the only one with this problem, judging by this thread on the Kaspersky support forums. Kaspersky promised to release an update that fixes the problem later on today.

Update October 25th, 2010 - 14:47
*phew* The update seems to have solved the problem.

It’s been a while since I decided to rebuild my website from scratch. I’ve been really busy with other stuff, but I finally found some time to make a simple photo album system.

Not all albums are visible yet, which is due to a conversion mistake from the old system. The missing albums are still intact, but the timestamps have been messed up. As soon as I’ve figured out how to fix that, those albums will become visible as well.

As promised; you - the visitor to my website - can now post comments on my blog entries. It probably needs some tweaking here and there to keep the spam bots out, but it is working as it should be at the moment.

You may also have noticed that the photos are working as well since a few days. Good grief, it might actually turn out all-right with this self-made code

As you can see, the blog is more or less back online. It’s still in a very primitive phase, and lots of stuff still isn’t working. On the top of the to-do list are:

• The ability to add comments on a blog post.
• The ability to search in the text of the posts.
• The ability to see photo’s in the posts again (Photo’s aren’t working yet, video’s however are).
• Etc...

Once those things are working again, more modules (such as FAQ, Projects and Downloads) will be added.

On the topic of video’s... I’ve made some changes to the system for the videos that result in much higher quality. The old videos were encoded in Adobe’s Flash Video format at 15 frames per second and used up 512 KBps. The old videos have been resized and re-encoded in H.264/MPEG-4 format at 512KBps. The difference is very noticeable, I think. New videos (such as the videos I shot at the Amphi Festival) are encoded in H.264/MPEG-4 format at 30 frames per second and use up 1000 KBps.

I’m all set. This next Saturday (July 25th, 2009), I will be moving my new web server from the “staging area” (read: my bedroom) to the data center in Amsterdam. Sunday (July 26th, 2009) the old server in Canada will be powered down and dismantled.

Aside from departing from the server in Canada, I will also be departing from Xoops (the CMS that I’ve been using for 2 years now). I’ve decided to write my own website code, for a couple of reasons: security, speed and size (also related to speed I guess…).

The more code you have, the slower a site is, and the more can go wrong. Xoops is a very large CMS, with a lot of functionality (most of which I don’t use). If I write a minimalistic CMS myself, with just the things that I use; it should - in theory - make the site smaller, faster and more secure.

So… this Saturday my server will go online, but my website will be offline for a while until I’ve made a basic blog module.

Wish me luck!

Today, my black magic woman arrived. Unfortunately, it turns out that the 2x1GB memory I had arranged doesn’t fit. But with the 1GB that was in the server and 4GB extra... it’s still 5 GB and that’s still plenty for a webserver.

Some snapshots for your viewing pleasure:

*purr*

As some of you might, or might not know, I rent a dedicated server at iWeb in Ontario, Canada. They have some decent deals going on and starting at USD 69,- a month you can rent your own server. That is… if you stick to 1GB ram and either Linux or FreeBSD. If you want a different operating system, you have to pay extra, and if you want more memory, you also have to pay… extra. In my case it ended up at USD 109 a month for An Intel Celeron D 3.0 GHz with 2GB Ram and 300 GB IDE hard disk, equipped with Windows 2003 Standard Edition. The package includes a 10MBps uplink and 1 TB of traffic per month.

This is an average price for renting a dedicated server, but it always struck me as odd that I have to pay USD 10,- a month for an 1 GB memory module that costs € 15,- at my workplace. Time for change, I thought… so I did a bit of researching and found out that for € 49,- per month, I can collocate my own server on a 100MBps uplink with 1 TB traffic per month, at Trans|ip, the same company where I have my domains registered. It’s not rocket science to see that it would save me about € 35,- to € 50,- per month, depending on the exchange rate of course.

I would get 10 times the uplink speed for less money. The only problem is that I didn’t have my own server. I had already decided that I wanted a certain minimal configuration:

• Dual Core processor
• 4 GB memory
• 2 Hard Disks in raid 1 configuration

First I checked the website of my employer, Aces Direct, of course. Unfortunately, the cheapest server that met my desired specifications was way over the budget that I had in mind. The problem is that most servers are sold without disks, and server hard disks are a bit more expensive than consumer hard disks.

Next I ended up at a company that sells reasonably cheap web servers. For € 399,- (ex taxes of course) you can get a simple server with either an AMD Athlon 64, AMD Sempron 64, or Intel Dual Core E2220 processor. A bit of research told me that neither of the AMD’s was Dual Core, and that the Intel was a first generation Dual Core processor and had performance that was comparable to a single core processor.

A co-worker told me that one of our suppliers might have something on stock. On our website, we prefer to sell the latest models obviously, but the suppliers might have an older model on the shelf somewhere. A quick E-mail here and a phone call there told me that indeed one of our suppliers had some older models on the shelf that might fit within my budget. The price would be comparable to the cheap web server with the AMD or Intel Dual Core processor, but it would be a Hewlett Packard or I.B.M. Of course I feared that it would end up way too pricy again due to the more expensive hard disks, but the supplier told me not to worry about it.

A bit of haggling and ass kissing later, I had made a very nice deal on my new server, which should arrive next Monday or Tuesday. I went a bit over my planned budget, but also managed to make a deal with my boss so that I can pay for the server in parts. I’ll pay half of the server in cash (which is well within my budget), and half of it will be deducted from my salary in 3 parts. So what did I get?

From a supplier I managed to get For € 819,91 inc taxes :

• Hewlett Packard Proliant DL 320 G5P with Intel Xeon 2.66 GHz processor.
• 4 GB memory upgrade kit.
• 2x 250 GB hard disk (hot plug)

Via another channel I also managed to get a 64-Bit Windows 2008 Web Edition license and an additional 2GB of memory for free (the guy owed me a favor :P). I think going from a Celeron D 3GHz with 2GB ram and IDE hard disk to a Dual Core Xeon 2.66GHz with 6 GB ram and raid 1 hard disks is quite a nice upgrade, and after the server has been paid off, I will save money and have more performance.

My Server should arrive Monday or Tuesday, so stay tuned!