Working on the mysql server crash issue

[Steve St.Laurent]

I'm monitoring the site closely tonight. Periodically during peak periods something is happening that causes Mysql to go into a runaway situation and then it refuses further connections and I have to reboot the server. The problem is that once it's locked up I can't see what was happening just prior to the lockup to know what's causing it. I'm working on some tools right now to collect data building up to the lockup. I will be monitoring the site closely tonight - if you get a database error check back in a couple minutes because I should have it back up momentarilly.



-Steve

 

[Tardog]

Once again you are very hard at work behind the scenes. Always, appreciated. Many thanks,

 

[mberry]

Thank you for all the hard work Steve. I would help if I was there. If nothing else, I could fetch the coffee.

 

[Steve St.Laurent]

I got 16 hours of sleep last night - went to bed @ 11:30 pm and didn't open my eyes until 3:30 this afternoon so I don't need the coffee too bad . I REALLY needed the sleep.

 

[BKenny]

Yeah, keep up the good work Im very good at this computer stuff i still can,t post pictures even the instructions confuse me :-laf

 

[HerbW]

Keep up the good work!

Steve , We appreciate your efforts. Thanks, HerbW

 

[BushWakr - RIP]

I agree with the rest... . thanks for ALL the work you do on this site... . lord know's I couldn't do it or find the time... . good on ya Steve !!!!



Bob...

 

[DPelletier]

I got 16 hours of sleep last night - went to bed @ 11:30 pm and didn't open my eyes until 3:30 this afternoon so I don't need the coffee too bad . I REALLY needed the sleep.

Glad to hear it Steve! My take on the situation is that as much as I enjoy this site, it being down for a little while is a small price to pay for you to have a decent nights sleep! Don't overdo it, bud!



Cheers,

Dave

 

[cmills]

thanks

I agree, thanks for all the long hours of hard work

 

[AMink]

Ditto! Oops, I hope that's not a copywrited word:-laf Thanks for the work Steve.

 

[DieselMinded]

Sorry! The administrator has specified that you can only do one search every 15 seconds..... LOL



Im trying steve



DM

 

[Steve St.Laurent]

Well, I've found the problem and I think I've found the solution. Mysql was crashing because it was running out of memory - except that it wasn't. There was a meg and a half of free memory when it would crash. When the CPU load would hit 99% when lots of searches were going on it would crash out. I was able to replicate the problem by doing 4 or 5 searches simultaneously from my system. Being able to replicate the problem is 80% of the battle. I couldn't find any reason that it should be running out of memory so I tried upgrading to a more recent (the most recent stable) version of mysql (was on 4. 0. 17 - now on 4. 0. 22) and had users hit it with as much traffic as we could muster (I had 11 searches going on, had a couple other chat room users running 8 others, plus whatever the online users were able to get done - thanks all) and couldn't get the load over 92% and it never crashed. I will continue monitoring it tomorrow but hopefully this one is now solved.

 

[BadDNA]

Excellent job, thanks for all the effort Steve!

 

[Steve St.Laurent]

Well, nevermind. There's still a problem - I just couldn't create enough traffic last night to make it come up. At least I can still replicate the issue. Back to the drawing board . . .

 

[JR2]

There seems to have been a slight problem with the TDR Roundtable database.

Please try again by pressing the refresh button in your browser.



An E-Mail has been dispatched to our Technical Staff, who you can also contact if the problem persists.



We apologise for any inconvenience.

Is that what this is??

 

[Steve St.Laurent]

Yes it is. I'm still working on it.

 

[Steve St.Laurent]

I've made some major changes to the search engine today. The vBulletin search engine is what has been causing the crashes. I found a number of other large sites that had to trash their search engine. There is a hack that uses mysql directory for searches and they've had good luck with it. I have switched the software over to that. I will be monitoring the site closely tonight. If you get any database errors tonight please come back in a few minutes and I should have it back up.

 

[TBohrer]

Seat belt light

Hi, Yesterday my seat belt light on my 99 Turbo started staying on. My seat belt is fastened and no one else is in the truck. Has any body else had this problem.

 

[Steve St.Laurent]

So far so good (I'm probably going to kick myself for this later tonight). We are running at a query rate of 192,066 per hour which is getting close to our peaks and in the last half hour (since I last rebooted mysql) we have had ZERO slow queries. Searches seem to be smokin fast for me thus far. We tried to load up the server with queries to get it to crash (which we were able to do this morning no problem) and the load hardly even moved. Keeping my fingers crossed.

 

[DGFoster]

Good work Steve