The servers are working great & search info!!

[Steve St.Laurent]

Great news! The servers are working better than they ever have since I took over the webmaster job with the latest change. More on that later



We have outgrown the search capabilites of vBulletin's search software and that is what has been causing the crashes over the latest several days. I wrote a number of scripts to log data building up to the crashes so that I could figure out what was happening because once it would crash I had no way of pulling information out as to what had happened. Night before last I was finally able to nail down what was causing the crashes and was able to replicate the problem. During peak traffic periods I could start up 10 search windows and run those searches and lock up the server. During less than peak periods I would need some help from the other moderators, tdr staff, and users on the website (thanks to all of those that helped me crash it! ). I thought I had the problem nailed down last night but once the traffic started picking up in the morning and I was able to get enough people on to load it up we were able to crash it again.



At 11am this morning I went to bed to get a little rest and got 4 hours of sleep and then went back at it. I found a hack that throws out vBulletin's search engine and uses mysql directly for doing searches. There were several webmasters of large sites such as ours that were using it with success. The vB search engine works great for smaller sites that are resource bound (memory & cpu) but once the database gets to a certain size (we have close to a million posts) it becomes inefficient. It still should not have locked up mysql but that is what it was doing - even though it wasn't using anywhere close to the resources that I was allocating it. By getting rid of their search engine and using mysql direct it's now able to utilize all of the resources I'm throwing at it - which is an entire dual cpu system with 2gb of ram, a raid 5+1 disk array with a 128mb caching controller!



I had the site down for about an hour and a half today while I implemented the change. I last booted mysql up 8 1/2 hours ago after modifying the search engine. I got it done before our peak traffic period so I was able to adequately test it. I tried as hard as I could to up the load with the help of others as well and we could budge it! The best I was ever able to get the slow query rate to was . 006% and more recently it was in the . 040% range with 34 slow queries average per hour (out of 160,000 or so) - and that's an average including the slower times of the day. Tonight in the last 8 1/2 hours the server has processed 1. 4 million queries with ZERO NADA ZIP slow queries!!!! We hit a peak rate of 220,000 queries per hour. The database server load at it's very highest hit . 45 with 2. 0 being fully loaded - or less than 25%!



The web page server hit a load of 1. 57 for one 5 minute period (the log file writes every 5 minutes and it writes a 1 minute, 5 minute, and 10 minute average at that time) - outside of that it never went over 1. 0 (or 50% load). The server loads have really surprised me because I thought the database was putting a bigger load on the system than the web pages were but have found that isn't the case. Based on our current traffic growth we will need to add another web server in the not too distant future (bringing us up to a total of 3 servers running the site) and am making plans for that now.



Now for the only bad news in this whole thing - when we moved to vB3 I was able to turn on 2 word searches for some limited terms (vp, ve, lp, bd, etc, etc). With this change I don't believe it will be possible to have 2 word searches. Once I know this is all stable and running good I will look into it further but I doubt that is going to change. I do apologize for any inconvenience that will cause but I simply don't have any other choice at this point - there were no other options.



I'm really looking forward to things getting back to normal around here and being able to sleep like a normal person again. I'm 99. 9% sure that we're over the hump on this stuff. I feel 1000% better than I did 12 hours ago.



-Steve St. Laurent

Webmaster

 

[rweis]

Posted at 04:09 am huh, but seems like you found the problem. Time will tell. My hat is off to you, sometimes very tough, but at the same time very rewarding. Many times I wonder why in the world we get into careers like this, and doctors think they have weird long hours, lol.



Bob Weis

 

[BarryG]

Awesome job Steve, thanks for the hours and effort it is appreciated

 

[ProMac1K]

Not to upset your success Steve, but I don't know about the searching. I can't seem to search for anything without either getting 0 entries or 13, 17, or 19 pages of entries, no matter if I select certain parts of the forums or not. Searching is also basically the only reason why I subscribe, I don't start very many threads unless I can't find what I need in the archives, which was the case tonight because of the search results. How come I am getting such bad results lately, since the site has been fixed? I don't want to search through how many pages of threads, plus how many posts in each thread, for what I want. That is what searching is for.

 

[John - K5AWO]

Steve,

I do believe that you have done it! The web pages are zinging in here on my laptop thru a wireless hub. Faster than I have ever experienced.



Just for grins, I did a search for "fan clutch" and before I could say "One Thousand and one" it told me there were 287 threads on that subject and the page was fully displayed too!!!



An awesome job Steve... Kudoes on a superior job performance rating.



John

 

[John - K5AWO]

And to add one more comment. I have been experiencing long waits when a message was posted before it displayed my post.



Well, it came back this time before I could get my fingers off the ALT-S keys... .



Fantastic

 

[DEickhoff]

Thanks for all the hard work Steve. You've done a great job!!

 

[Steve St.Laurent]

Tmaas - What were you trying to search for? I've done tons of search and come up with tons of threads for those searches. I've literally done hundreds of searches since the changes. If you could give me the terms you are searching for and the other options you are using (like what forums, show as posts, show as threads, how far back, etc) I'll see what's going on for you.

 

[ProMac1K]

Thanks Steve!



I do mostly multi-word searches so maybe this doesn't help, but it should help to narrow down the returned topics while searching.



Just a few examples here off the top of my head.



I was searching for idle adjustment in relation to the tachometer. I typed in -idle tach-, in all forums, and came up with 14 pages. Then I did a search with -"idle tach"- to shorten the returned entries, in all forums, and came up with 0 pages.



Then I did some searching to see what transmission was viewed as the best for pulling out people that were stuck in mud, or snow, or just for use in mud or snow. The first search that I did was -auto stuck-, all forums, came up with 19 pages. Then I did a search for -"auto stuck"-, all foums, 0 pages. Then I did a search for -auto pulling-, all foums, 18 pages. Awful lot of info to look through. Then I did a search for -manual winter- (searching for manual use in winter), all forums, 17 pages. Then I did a search for -"manual winter"-, all forums, 0 pages. Then I did a search for -"manual snow"- (searching for manual use in snow), all forums, 0 pages. Then I did a search for -manual snow-, all forums, 14 pages. Then I did a search for -manual snow- in the 2nd Generation Topics forum, 16 pages. How can narrowing a search down from all forums to just 2nd Generation forums yield more topics?



I did more searching but I couldn't duplicate them today since I cannot remember all of them. I had to re-search for everything to find the results, to post here.



It just seems that ever since this database error problem, searches have been either yielding a lot of results, or no results.

 

[ProMac1K]

Also how come I never got a email notification after you replied?

 

[Paychk]

Steve when I search on "fuel solenoid ohm" it returns a great many hits, but not one has ohm in it. The only option I change on the search page is to the "show results as" option and I make that "posts".

 

[Steve St.Laurent]

TMaas - I looked at those searches and in all cases it did exactly what you were looking for. "auto stuck" "manual winter" "manual snow" all came up with zero results because there are no threads that have those words in exactly that pattern. ie - if you searched for "auto stuck" and there was a post with this in it:

I have an auto transmission and got stuck

it would not return that as a hit. If there was a post with this in it:

Today my auto stuck in gear

it would return that as a hit because it's an exact match. Anything within quotes has to be an exact match to return a result.



The reason you may have gotten more results by searching just the 2nd gen forums is because the search engine will return a maximum of 500 results. Those 500 results is post based so the number of threads that can return can vary. ie - you have a thread that contains the word "auto" in it 3 times - that's going to count as 3 of those 500 results so now the most that could be returned is 497. By varying what forums you are searching it varies which posts are in that search hence the variance in the results.



Your emails are being sent and the logs show that they were received. Your ISP probably has a spam filter and is capturing TDR mail as spam. See if you have a spam folder and look in there - I bet you'll find them. You should be able to specify those as messages you wish to receive.

 

[Steve St.Laurent]

Paychk - there are additional modifiers that I need to add a help note to the search engine page because it is now a direct Mysql query. I didn't have time to get that done last week (plus I wanted to be sure it worked before I did it) because I was out of town since friday for a family wedding. I didn't realize this but when you do multiple word searches it will return the results of any of those 3 words unless you put a + in front of the word. So searching on "fuel solenoid ohm" (but without the quotes) will return any threads with any of those three of those words in it, searching on "+fuel +solenoid +ohm" will return threads with all 3 words in it. Here is the short help file they provided with the hack as to the allowable modifiers:



Allowed modifiers + are ,-, *, >, < and "

All modifiers except * should be used only once for one word (in the beginning and without space).

* it should be used at the end of a word.

For example:

windows unix - will find messages containing at least one these words.

+windows +unix - will find messages with both this words.

indows* will find "windows", "windowss", "windowssauce", è "windowst".

*indows will NOT find "windows"

"some words" will find "some words of wisdom", but will not find "some extra words".



This is good stuff because it gives us a lot more options on searching than we had before. I just have to get the time to put the options into a concise explanation so people can understand them.

 

[Yo Hoot]

Wow Steve... . right now it's running like lightning!

 

[Paychk]

Steve YOU DA MAN!



Thanks for all you do for us. This thing is running like greased snot Oo.

 

[Steve St.Laurent]

Here's a page I just did with information on how to use the new search engine - https://www.turbodieselregister.com/searchhelp.html

 

[ProMac1K]

Thanks Steve! Those new modifiers seem to work great. Really helps to find what you're searching for. Thanks for bringing those to my attention.



I don't think my SPAM system is blocking these notifications, because I got one for this thread today, plus three other notifications. Don't know what to say. I didn't get one yesterday though.



Thanks



-Trever