Default search date limit changed

[Steve St.Laurent]

To ease the load on the database server I've changed the default search date limits to the previous 3 months instead of any date. The reduces the number of threads that are being searched by default by an immense amount. If your looking for stuff further back than that you just need to change the "find posts from" on the search form to the range your looking for.

 

[rbattelle]

Well, that explains the suspiciously small number of hits I got on a search a few minutes ago! Thanks for the heads up, Steve.

Ryan

 

[Steve St.Laurent]

I figure it's easy enough to change and on many searches 3 months will give you 500 hits anyways (which is the limit), so having it hit the entire database on each and every search doesn't make much sense. Hopefully the search engine that is in beta will soon be ready and I'll be able to implement that which will be FAR more effiicient. Right now some searches are taking upwards of 60 seconds and that's on a dual cpu, 2gb ram, caching raid controller server that is doing nothing other than dealing with searches. The biggest problem is the way they work the database is locked until the search is done so 60 seconds worth of searches get backed up when that happens. By limiting them like this far fewer will take that long.

 

[CTD12V]

Hey Steve,



Does this default 3 month limit on the searchses trump the "save search settings" option? I tried saving my settings to include any time like the old version, but every time I need to modify my search query, it resets me to 3 months. In 99. 9% of my searches I want to search the entire history of posts, and having to reset the time limit every time I modify a query string is driving me NUTS.



Thanks!



CTD12V

 

[Steve St.Laurent]

For now yes it does, sorry. I would ask that you try doing a search on a shorter period of time before doing it on the full database. Herein lies the problem. If you do a search on the entire database no matter how many results there are a maximum of 500 results will be returned. If there are 500 results in the first 3 months and 3500 in the rest of the database it will still have to crunch through the entire database to return your search results. That search on the first 3 months may take 10 seconds whereas searching on the entire database will take 60+ seconds - and you'll get the exact same results because a max of 500 results will be returned. During those 60 seconds while it is searching anyone else trying a search has to wait until your 60 second search is done before their's even starts. You can see how this could bring the server to it's knees. I am watching closely for a production release of the search engine that is in beta and I am also looking at the feasibility of rewriting the search engine from the previous version that I was using for the new version. I wasn't aware that this issue was going to come up until the new version went live and I saw the server loads go crazy. I'll do my best to correct the situation as quickly as possible and I would ask that in the meantime you try to keep your search date ranges to as small a range as possible that will still get your results to you.

Thanks,
Steve

 

[rbattelle]

Are guests allowed to search? If so, you might shut that off.

Ryan

 

[Steve St.Laurent]

No, guests cannot search - only members can.

 

[rbattelle]

I had to run a couple big searches this morning, and it pained me to do so.

One thing I've always thought would be useful is the ability to restrict searches to only threads to which you are subscribed. With my post count I've got tons of thread subscriptions, and probably 75% of the searches I run are looking for things to which I subscribe.

If I could limit the search engine to search only my subscribed threads, that would significantly cut down on the load.

Ryan

 

[CTD12V]

For now yes it does, sorry. I would ask that you try doing a search on a shorter period of time before doing it on the full database. Herein lies the problem. If you do a search on the entire database no matter how many results there are a maximum of 500 results will be returned. If there are 500 results in the first 3 months and 3500 in the rest of the database it will still have to crunch through the entire database to return your search results. . .

I see. Well here's hoping you find a good fix soon! Thanks for all your work you put into this site.



CTD12V

 

[Willys]

I had to run a couple big searches this morning, and it pained me to do so.



One thing I've always thought would be useful is the ability to restrict searches to only threads to which you are subscribed. With my post count I've got tons of thread subscriptions, and probably 75% of the searches I run are looking for things to which I subscribe.



If I could limit the search engine to search only my subscribed threads, that would significantly cut down on the load.



Ryan

Try searching with your username in the 'search by username' field to the right. This should help unless you make a habit of subscribing to threads you don't post in. This is actually how I ended up in this thread- I am searching for one of my own old threads and was limited to 3 months of posts. I didn't think to check the date range of searches.

 

[rbattelle]

Try searching with your username in the 'search by username' field to the right. This should help unless you make a habit of subscribing to threads you don't post in. This is actually how I ended up in this thread- I am searching for one of my own old threads and was limited to 3 months of posts. I didn't think to check the date range of searches.

Yup! You're right. I've done that in the past and forgot all about it.



Ryan

 

[Steve St.Laurent]

I've rewritten the search engine so that searches are running fast again. So the search date range has been changed back to any date again.

 

[CTD12V]

Great!! Thanks!