The HIPComix forums (and some of the main site) suffered a massive crash Wednesday. Anyone who registered on the forums in the past week or so (which I think includes many of you who frequent this site) may need to sign up again. If you find that you cannot login to the forums, you can re-register using the same nickname and password because you old one was was deleted in the server failure.

At last check, the pay portion of the site is up and running. Obviously paying customers are our first priority. Anyone with a membership should be able to log in with no problems. Some of the images may be astray as we work to load all of the backup files. If you see anything out of place, please feel free to notify us here via the forums.

Jpeger will be along to give a more detailed and technical explanation. Thanks in advance for your patience and understanding.

Well, I don't know if I can tell anymore than what Doc just said.

Over the past few weeks, the web server has been having problems with corrupting data and files not writing correctly. We've run repairs on it, but I was concentrating more on software issues since it only appeared to be happening here at HIP.

Yesterday morning I logged in to look over the avatar issue, and noticed the dir structure was corrupt. I phoned it in to the hosting company I co-locate at. They began looking into the databases as if they were corrupted. The base was again repaired, and the system put back on line. By noon, the forums crashed again, and I called in a new repair ticket.

The system went off line at 12:20 for diagnostics. I got a phone call at 2:30 that the drive had failed, and was being replaced. At 5pm I got the call that the new server was up and the back-ups were mounted as a device.

Full back-ups were done every Sunday, and changed files were done hourly. Since the drive was slowly corrupting, I had to go through each back-up one by one, directory by directory and see which one appeared not to be corrupted. As you can see, that took most of yesterday and today.

On top of that, because I use apt-get (The debian equiv to windowsupdate) I was installing all the newest versions of apps.. including the kernel, so it wasn't as simple as just copying over the config files. I had to reconfigure everything from scratch.

It was a long and hard 48 hours, but it's appearing like everything is working. Sorry for the inconvenience. Sometimes backing up just isn't enough.

Well, I for one want to publicly thank Jpeger for his hard work in getting the site back up and running again. In retrospect, a failure that severe could have been a hell of a lot more catastrophic to the site, both in terms of lost content and amount of time offline. The fact that we were only down for a few hours is by no means a good thing but it certainly could have been worse.

I am sad to see the effect that the crash had on the forums. We had some good momentum and a number of fun topics going before many of our members lost all of their posts and even their profiles. I hope that perhaps after the holidays we can get back in the swing of things and get the discussions back to where they were just a few days ago, if not even better.