MySQL upgrade done on main server - keeping fingers crossed ;)

[Steve St.Laurent]

Well, I just finished getting the MySQL upgrade done on the primary file server. We've had a couple things hit us in the last couple weeks. The slowdowns were primarily caused by a lack of Ram which was remedied on friday. With the Ram upgrade however we were able to load MySQL up to a point where we uncovered a MySQL bug! The database crashed twice last night while I was monitoring the system (been monitoring it closely all weekend waiting for it to crash) - so I was able to see exactly what was happening. I spent all night working on the server and installing the MySQL upgrade - which entailed upgrading about 30 other packages as well. I now have to get the slave server upgraded and then we'll be back up 100%. As a function of the upgrade I also had to start over with the configuration of MySQL (because there's a number of new options) so I will have to do some tweaking to the settings as we load the server up. Until we get the load on the server I won't be able to tell what needs to be tweaked - because this version does some things so differently. I will try and monitor the site during the day today (although I do plan on trying to get some sleep - I've been up for 28 hours).



-Steve St. Laurent

Webmaster

 

[Gray Ghost]

Know the feeling. Did that for years on mainframes. Loved the MG's on IBM machines. Nice and warm and you could get some good ZZZ after being up 30 to 40 hrs. Can't do that with a file server.

 

[Dean Upson]

Thanks, Steve.



I don't know much about what you are talking about but certainly appreciate your hard work. I know what long hours are, get some rest and stay away from the coffee, soda and chocolate for a while



Dean

 

[camnicolson]

Don't know what all that means , but hey, good job and enjoy your sleep



Cheers,

Cam

 

[Gary - K7GLD]

God, how I hated all the acronyms used in computer system operation/management... As a former system administrator at the newspaper I retired from, first on mainframes, then later on desktop units as they became more rubust and common, I had to know them all.



For the uninitiated, let me explain the "mySQL" bit - it's a little easier than some.



As it sounds like, it refers to "my squirrel" - smaller, less powerful file servers only use chipmunks for power - those are what you normally see over in the Ford and GM forums - but our more powerful server here on the TDR has upgraded to squirrel power - which as can be visualized, consists of a squirrel running inside a wheel that powers the file server. Problems arise when the squirrel gets tired and slows down, gets distracted, or possibly dies and has to be replaced with a new healthier one.



So now, you have a clearer picture of our problem - we have a sick, tired or distracted squirrel - and Steve is having to provide mout-to-mouth to get him back into normal operation...



I'm always glad to help others out in understanding these more difficult concepts...

 

[Tony T.]

I know exactly what he's talking about WHERE DOOMED! It had to be a reoccurring echo from Y2K that crashed the system. Maybe from the Mars probe landing. The machines are taking over. This sleep excuse was just to pacifier you all, while in reality Steve is busting his butt at this moment to save the system from catastrophic destruction. I for one am taking no chances and heading for the shelter. Steve Good Luck and God's Speed. :-laf P. S. Steve Your doing a great job!

 

[Dean Upson]

Originally posted by Tony T.

I know exactly what he's talking about WHERE DOOMED! It had to be a reoccurring echo from Y2K that crashed the system. Maybe from the Mars probe landing. The machines are taking over. This sleep excuse was just to pacifier you all, while in reality Steve is busting his butt at this moment to save the system from catastrophic destruction. I for one am taking no chances and heading for the shelter. Steve Good Luck and God's Speed. :-laf P. S. Steve Your doing a great job!

After reading Gary's post, I will take TonyT's advice, join the squirrel, grab my nuts and run for the shelter. Oo. Oo. Thanks again Steve.



Dean

 

[Steve St.Laurent]

Just woke up - man am I tired. There are currently problems with the classifieds and with the travel companion (that I am aware of). In order to fix both of those I have to recompile Apache which will take the site down while doing the compiling, etc. I'm going to wait until late tonight (after midnight) to take the site down to do that to minimize the disruption.

 

[Steve St.Laurent]

I have to leave the house for a couple hours. Robin's watching the site and will let me know if there are any problems. Good news is that so far the performance seems great. We've had over 500 users online and our slow query rate (queries that take more than 10 seconds) is at . 006% (12. 398 slow queries out of 224,404 per hour) which was at or about . 020 the last couple weeks. Our server load hasn't gone over 1. 3 (it was spiking up to 3 and 4 before). So far things are looking great other than the couple of software issues - which I expected some would come up. Hard not to have a couple bugs when you've been awake for 30+ hours and recompiled ~30 software packages. Just the downloading of the packages took over 3 hours on the servers high speed connection!

 

[lil red cummins]

i really appreciate all your hard work steve. and all this while you are healing. what else can i say. this is an awesome site with awesome people.



jim

 

[FFL8N]

I also would like to thank you personally! You are going over and beyond what should be expected from you! THANK YOU!!

David

 

[jcarew]

Good Job I don,t know computer jargon but I do know your handling of this site is great an as for Garys squirrel stories feedum corn they like that. Thank You, Get Some Rest

 

[TJMIV]

:--) Hey ,did ya`ll get all that! Steve I`m glad you know what your doing, cause I sure as heck don`t . I just wanted to let you know that this is a great web-site because of people like you & I appreciate the time & effort. I kind of agree with Dean Upson on this computer thing cause I can barely get online.

 

[RottonRob]

Thanks Steve for all your hard work.

 

[Deezul1]

Great job Steve, I know where the lift pump goes but after that i am lost and glad you can do the rest of it.

 

[rweis]

I do the NA for a small central florida university. Can be FUN? at times. Been there, doing that.



Have what suffices for fun? in our business, better you than me, lol.



Great site, runs really well for the load we put on it.



Like the CTD, what size injectors did you use this time? Just a minor fuel leak. No problem.



Really does run well though.



Bob Weis



Network Administrator

Webber International University

 

[Steve St.Laurent]

Ok, everything should be back up and running 100%. If you have any problems let me know. The site ran great tonight with record traffic levels. We were transfering over 2mb/sec for a while tonight with a new record of 661 users online. We've never been over 2mb/sec up until now. Didn't see any problems on the server at all. We never went beyond 65% of the memory used, server load never went over 1. 42, and our slow query rate stayed at . 005% (we were over . 020% the last couple weeks). So far I haven't had to change any of the MySQL settings from how I set them up initially.

 

[phoekema]

Ditto's on the good work, Steve. I was in charge of a Unisys mainframe for a few years along with their DMS database system. Between full backups, audit file backups, restoring when a drive went down (more often in the old days than now) and data base reorganizations, it was enough to drive you crazy. It has been said one of the advantages of being in computers and data processing is when you go nuts, no one will notice the minor change!!!

The most frustrating part is you very carefully plan a database reorganization for a weekend. You do your best to take in all possibilities and conditions, but more often than not, at 2 am on Sunday morning something happens you never planned on. Blood presure higher than a common rail fuel system!!!

I work with pc's now, but mowing lawns and cleaning windows (non Microsoft, of course) looks pretty good some days.

 

[Gary - K7GLD]

N/A can be VERY stressful at times - epecially in a time (deadline)-sensitive, dollar intensive environment - nothing like sitting on a computer room floor surrounded by various electronic components as you scavenge parts from one power supply or hard drive to get another up and running - while 60 people are walking the floor and watching the clock, not knowing whether or not we're gonna get paper out that day or not. .



Do the words Control Data Hawk drive mean anything to anyone here?



Yeah, like one fella said above, MANY times mowing lawns for a living looked REAL attractive!



And another old saying - "of all the things I miss as I grow older, I miss my mind the most"...



Good job Steve - I don't envy you in the least - sometimes (many!), the money just isn't worth it!

 

[Tony T.]

Steve

Is it safe to come out of the shealter yet!:--)