Why the forum was EXTREMELY slow and then down tonight (08/08 in the a.m.)

[Steve St.Laurent]

I discovered a problem last night with the vB database. The number of posts and threads shown in the forums did (does) not match up with the number of records in the databases. I found this out by looking at the stats on the front page which goes by the number of records in the database and comparing them to what the forums showed at the bottom. After looking at the database I found a bunch of really old threads that were attached to forums that did not exist and had a copy in a forum that did exist. There were approximately 18,000 of these. These threads were from a time period before I took over the webmaster job. It was probably a bug in one of the previous versions of vB that created these.



There is an "orphan thread" routine that goes through and cleans things out so I attempted to run that. Well, running that routine brought the site to it's knees because of the number of queries it ran. Once I figured out what was going on (about an hour worth) I shut down the forums to try to get this done as fast as possible. What I found was that because those posts were in the search engine it was going to take ~48 hours to delete all those threads. Because of this I had to wipe out the search engine and start that process over again. So, the BAD news is that it will now take about a week to process all of the posts and put them into the search engine and the site will be slower than normal during that entire process .



There are still some issues with the database that I will be contacting vB about because there are 30 posts that aren't being recognized by the forum software and ~3,000 threads also not being recognized by the software that are in the database. This situation has been with us all along I am quite sure. The old version of vB compiled it's stats differently (the same as the stats on the front page) and that's why I never noticed it until now.



I've been pouring through data for the last 5 hours and I can't keep my eyes open any longer. Thus far I haven't been able to figure out a pattern or even which records aren't being shown (although I've found some). Once I get some rest and can look at the data with a fresh mind I'll figure out what's going on and get ahold of vB to see what can be done about it - if anything even needs to be done.



Sorry for any inconvenience tonight caused you and I'm really sorry that the search engine rebuild is having to be started all over again. The good news is that the database is a lot cleaner now than it was 5 hours ago which will help with mysql performance.



I'm going to bed . . . .



-Steve St. Laurent

Webmaster

 

[Russknh]

Steve, I know you have been told before but a guy can use a little reminder every once in a while---THANKS FOR ALL YOUR WORK AND THIS BOARD !!!

 

[John - K5AWO]

Amen!

I knew that the server was in trouble. Started to write a post about pages taking more than two minutes to come up, but by the time I got the post written, vB was really hung... .



Then a few minutes later I got the message that Steve put up for us and said to myself "Ah ha!" and called it a night. Try to get some sleep, for me that is, I knew that Steve wouldn't be getting any sleep until he figured out what was going on.



Said it before... . Steve, you are one dedicated son of a gun.



Kudoes to Steve

 

[JGK]

Steve,



You truly are a Saint. Stay with it. We can live with it until you get it straightened out. IT folks are always blamed when the system doesn't go right but are rarely rewarded when it works as expected.



JK

 

[DBoyd]

And I second that "AMEN"!! Thanks for all the work you do on this site. And I am really enjoying the upgrade.

Thanks again.

Don

 

[rrausch]

Ditto on what the above guys say!! Thanks Steve!

 

[Steve St.Laurent]

I'm working on this right now and some of the queries I am having to do are causing extreme loads because it's having to go through everything in the post database (900,000 records). Sorry for the inconvenience, I'll try to have it over ASAP.

 

[Steve St.Laurent]

Ok, it's all done now. The majority of the orphan posts came from the original conversion of data from ubb a few years ago and have been taking up space and resources since then. The database is all cleaned up now and the statistics are reading correctly. As of right now the search index rebuild is on post #248,000 of ~890,000. The search rebuild is going a little faster because of the clean up of the database but it will still take at least 4 more days to complete. The further along it gets the more it slows down so I'm never real sure how long it is going to take. It's possible that it could still take a week to a week and a half from last night to complete. I'm looking forward to getting a full nights sleep tonight!



BTW, thanks for the good words guys - it's appreciated.

 

[John - K5AWO]

Steve,



Don't be worrying over us out here. We can get along just fine. The server is running consistantly now with pages loading and ready to read in about 7-8 seconds. That ain't all that bad.



I just tried a couple of other vB 3. x boards

http://www.pirate4x4.com/forum/showthread.php?t=182536 loaded in 6 seconds.



http://www.dieselforum.org/portal/consumer.html loaded in 4 seconds.



http://www.nwbombers.com/cgi-bin/forum/ikonboard.cgi?;act=ST;f=3;t=8877 loaded in 3 seconds for text and 7 seconds for all images.



So, even with the search engine running wide open, this server is doing an outstanding job.



-John