Site maintenance issues - down time today (tuesday)

[Steve St.Laurent]

Well, THAT could have gone better! We had one of our 4 servers crash about 3 weeks ago and we had a consultant go in today to change out that server and at the same time have him check the other 3 servers for dust, that all the power supplies and hard drives were functioning, etc. I estimated it was going to take an hour. Well, when I tried to bring the site back up all kinds of problems surfaced. One we had 2 data cables on the private network (where the servers communicate between each other) that went bad in the process and also the user permissions database for the database server as well as the vbulletin database got corrupted in the shutdown/startup process. Took a while to get it all figured out as to what the problems were. Multiple simultaneous problems are difficult to debug. I don't think we lost any data at all. Sorry for all the down time but we should be all set for a while again. The 4th server will be coming back on line later tonight when I move the current database over to it. Pretty stressful afternoon.

 

[seafish]

THANKS to Steve AND his perseverance in the face of Murphy's Law

THANKS for putting all the effort into it. Of course, it's also a quick and dirty reminder about the infinite relevance of Murphy's Law and all of it's corollaries. #@$%!



My favorite corollary-- Murphys Law of Thermodynamics--

"Things get worse under pressure"



My second favorite-- Mrs. Murphy's Law

"If anything can go wrong, it will happen when Mr. Murphy is out of town. "



Here is a site full of them, for fun reading!!:-laf



Murphy Laws Site - Murphy Laws



Hope things go smoother tonight!!!

 

[Steve St.Laurent]

Gotta love Murphy. Well, the work tonight went off without a hitch. We're back up and running on all 4 servers again.

 

[PGAdams]

Steve,



Good work.

I have been in the telecom, data world for over 30+ yrs.

The one thing you could always count on was, it either worked or it didn't.

There was no faking it.

The other often referenced quote was "murphys law" as you experienced yesterday.

Murphy Laws Site - Murphy Laws