Updates on the server upgrade

[Steve St.Laurent]

As some of you are I'm sure aware there was a server crash this morning. It was on the web server (not the database server) and we're not sure exactly what caused it as there's nothing in the logs. It's possible that it is a hardware problem and we will be monitoring it very closely - rebooting the server brought the site back up. We tested the server as best we could and this is the first time it has locked up. If it happens again then we will go to our emergency plan and put the whole site on the database server while we figure out what's wrong with the web server. The way the system is set up if we lose one server we can reconfigure the other one to take over everything while it's fixed (they continuously back each other up) and based on the server loads we've seen either server could handle the entire site right now easily.



Now on to the good news, the servers are performing awesome as we expected - the load we are putting on them is hardly noticeable. While the site was down the webmaster also upgraded vBulletin to the latest version and there's lots of new features for us to play with (I'm finding new stuff all the time) - ENJOY! Searches are back up and running and you can now do 3 character searches with wildcards. We have also turned back on the ability to upload pictures to the server directly within a message rather than having to go through readers rigs to do it. In addition, the number of times a thread has been viewed has been enabled again on the forum screens.



There were quite a few bugs from the changeover and we've repaired all those that have been reported and found thus far. The old servers forums were not disabled immediately after the changeover and there were some posts put on there that won't make it over here unforetunately, sorry about that. As of now the forums on the old server are shut down completely so that won't happen any more. What that does mean is that during this transition period any links to old threads will not work - but they will work again in a couple of days when the site move is done. The links at the top of the forums were linking to the non functional pages on the new server (for now) and they have been fixed and are now pointing to the old server. Also, there were a number of graphic buttons/icons that were corrupted and those have been repaired as well. If you discover any bugs please report them to rpatton@ix.netcom.com and we will do our best to get them corrected ASAP. There are still some issues with the links at the top of the links, events, and travel companion screens and I will be working on those as soon as I'm done writing this. I just wanted to give everyone an update on where things stand.



The rest of the website should be moved over to the new servers in the next 48 hrs. Once everything is moved over then we will update the DNS entry and then the links to old threads will work again. Hopefully the web server going down was a fluke and won't happen again - but we have a plan if it does. THANKS for your continued patience during this server move! Everyone involved with the website is exhausted but we are making progress and it will all be over soon. Then we can get back to the job adding features to the site.



-Steve St. Laurent

Lead Moderator

 

[ZEEEW]

If you know what he is talking about, you are ahead of me

I only fix mechanical stuff

 

[DCMille290]

Great job Steve, I know I speak for many of us not so technically inclined, in saying thanks for all of your effort.

This is no small task or easy job to undertake, I've been there.

So, we are all waiting for everything to come together sooner or later.

A big hats off to all of those working on getting this project back on track and the site up and running with no glitches.



DC Miller

Marbleman

 

[cdpsrjc]

Steve,



Great effort on the upgrades. I'm particularly impressed that you have the reserve capacity on the database machine to run the website. Great planning. Things do happen with hardware, and everyone with a PC or CTD knows this. Please know we are pulling for you and appreciate the efforts. Take your time, diagnose it correctly, and TRY to have a good time. I'm a professional techie and believe me, you'll look in the mirror after some sleep and see a grinnin' face staring back at 'ya.



Richard

 

[KPhillips]

seems to be working great right now , thanks for the hard work. You have any idea why it won't save my profile info such as my location. etc. It keeps deleting it??? I just re-entered it, maybe it will stick now with the latest upgrades...



kerryp

 

[me4osu]

We may crack a lot of jokes and make smart--- remarks, but we really do appreciate the work you guys are doing.



In my limited IT knowledge, it sounds like the upgrades are great, and I know they are desperately needed.



THANKS!!

 

[jbone]

Don't know how you guys managed to keep it running with the crash and all, but good job... ... one question--when will you turn signatures back on, the sure help knowing how or where someone is, with upgrades... ... ... ... ... . R, J. B.

 

[Steve St.Laurent]

Well it's official, we're having problems with one of the servers (the main/web page server actually). There's a hardware issue with one of the servers. The secondary/database server wasn't quite completely set up to run the entire site so it took us a little while to handle shutting down the primary server and getting the secondary server up and running to handle the complete site (thanks to the web site team). That transition should happen very smoothly in the future if it should be necessary again. We are running completely on the database server (which is still way better than the server we were on 2 weeks ago). The primary server is being delivered to the manufacturer for them to figure out what's wrong with it and will be back online as soon as we receive it back. Due to the way the systems were set up we shouldn't have lost any posts and that was one of the goals of the new system. Thanks for your continued patience during this transition period. We tested the servers as best we could but you can never test completely (I've been in the PC hardware/software business for over 20 years).



-Steve

 

[Jim Fulmer]

You know it's not that you'll are bad. we are bad. I was online at 5:15 this morning and was angry that I could see what is going on, yep I'm a junkie for TDR..... Keep up the good work!

 

[wrk2mtch]

Steve,

Downtime happens. We all appreaciate the hard work you guys put into the site.



What kind of hardware are you using?

 

[Steve St.Laurent]

AHHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!! All I can say is that I hate Murphy! The server being down today wasn't anything wrong with the servers or anything on our part. The data center was upgrading their power system today and cut power to the UPS's as well as the power feed. That downed the server. The server case was locked and the data center people couldn't get into them to turn them back on. The keys were left on the back of the servers for them to have access but they are no longer there - where they went is anybodies guess. The entire data center was down for a while and there was also a backbone problem in the Atlanta area. My head hurts . . . . . .

 

[Steve St.Laurent]

BTW Terry, here's the hardware specs:



Rack mount case with extra fans

Dual AMD Athlon 1900+ MP processors

Tyan S2462 dual processor motherboard

2 GB ECC RAM (error correcting)

4 - 18. 2G 15,000 RPM Ultra 160 SCSI HD's (full RAID 5 array with 1 standby HD) - hot swappable

128MB Caching RAID 5 SCSI Controller

Dual hot swappable power supplies

Dual network cards

FreeBSD



The server's are smokin fast! So much so that we're putting a stress on the data centers air conditioning system in the room we're in and they have ordered new AC units to handle the heat our servers are generating. The two servers raised the temp in the room we're in by 10 degrees.

 

[wrk2mtch]

Very cool!!!



It's nice to see AMD getting a piece of the server market. What do you guys think of the Tyan board?

 

[pwknapp]

Great job!

 

[dseabaugh]

We also run Athlon. I couldn't wait to get one when they first came out! Just like the new 2003 Dodges. Way to go, Bomb the servers!!!Oo.

 

[DBoyd]

Many thanks to you for all the work you do to keep up a great site!! It is greatly appreciated by all of us!

Don

 

[Don M]

Symmetric multi-processor (SMP) systems are generally supported by FreeBSD, although in some cases, BIOS or motherboard bugs may generate some problems.



Don~

 

[Steve St.Laurent]

I sure wish Murphy would get off of our backs already! The room we are in at the hosting facility is getting hot due to the addition of our super servers. We're going to stay running on the one server until their A/C system is upgraded (parts are ordered they're just waiting for them to come in and be installed). Today we brought the server down to install exteremely high flow fans on the CPU's as a stop gap in the mean time. Those fans caused the server to lock up within 5 minutes of starting it up. In over 20 years in the business I've never seen CPU fans cause a machine to lock up - oh well . . . . . . . Because of that lockup the MySQL database was corrupted because it was not able to be shut down properly. We fixed the corruption a little after 6:00 pm eastern. Any posts that were put up between noon and 6:00 were lost due to the corruption so you'll have to repost those. Sorry for the inconvenience and again thanks for your patience during this troubling time.



-Steve

 

[Jumbo Jet]

I know this was discussed before and other options were already on order but -



Being the manager of a major corporation's 300+ Intel servers, we stick with the technology from the major vendors. Currently using IBM servers (have more Compaq and HP than IBM) that AUTOMATICALLY either speed up the CPU fans or slow them down as needed when the heat load increases. Also, if there is absolutely no way to cool the CPUs down with the fans, the CPUs will step down to slower processing speeds to internally cool down the CPUs. The last ditch effort to protect the equipment is to just shutdown the servers.



While all of these events are happening, the systems are notifying my administrators so that they can issue preventative measures.



These new systems are both faster and better. The administration efforts are still very important. Our guidelines show that the hardware is the least cost piece of the formula compared to software and system administration.



Isn't it just frustrating when some small things thwart the efforts of all the planning and thought that was put into your project?



Keep up the good work. It will all come together.

 

[Steve St.Laurent]

FYI - Tonight around 7:50 pm the site couldn't be reached for a few minutes, came back for a few minutes, and then went away for a few again. I was on the phone to the data center right away and they were making changes to the firewall and that caused the outage.



-Steve