slooooooooooow

[Yo Hoot]

I don't agree that it's the end user.



I see search times in the 30 second range and more every once in a while... . this is the number this site records on the top right. That's not an end user parameter.



Yes this is a paid site. It's also a very busy site. All I ask is that it remains available and works reasonably well.



That... . it does do.



On the other hand... we hope our dues are used to inhance the site performance when it becomes an issue. $35 x lots of members is an aweful lot of cash.



I have talked with Steve on a number of occasions. He seems to know what he's doing but I don't like always being told it's on my end.



I belong to enough sites and forums to know otherwise... unless this site has something very different going on that effects the end user's computer.



This is my only paid site and it is one of the slowest.



With that said... .



It's still a great site... . works ok... has many good features and isn't locked down much. Overall I like it and the way it's run.



I would expect some performance issues with such a large membership.

 

[Steve St.Laurent]

jrobinson2 - you can update your Java package at http://www.java.com . For dialup users I highly recommend trying firefox ( http://www.firefox.com ) as a browser - from everything I have heard it is FAR faster than IE or Netscape especially on dial up connections.



Yo hoot - The last thing I'm trying to say is that it's always on the users end. Look through my posts on this thread - on every one I've mentioned problems on our end and what I'm doing to work on those. That said there often are problems on the users end which can also improve things and has in many instances. I'm trying to do everything I can.

 

[WyattEarp]

Steve correct me if im out of line but seems to me all of a sudden this has become an issue in the past few days as from the posts i have read. This site has alot of features and along with the TDR magazine, and the server equipment to keep this site going is not cheap at all along with upgrades and software there is more than alot know to keep a site of this magnitude running effeciently, as i have posted before i still have dialup and every now and then i get the problems the rest of you are writing about but usually when i have problems it lies with my ISP provider and their servers and if your ISP provider has a system that can't handle the traffic coming online to the internet then your going to have problems with not only this site but others, I know that after killing alot of spyware that i didn't realize i had in my system this thing runs much much faster and there are some viruse's out there that not all antivirus software with detect and some of those virus's don't always cause instant problems, they lay and wait and gradually bog your system down. But this is just my 2 cents worth as im not a computer whiz but am pretty computer savy... . hopefully something i mentioned will help you guys out but keep a check on your pc systems as there is alot of junk out there in internet land just waiting to attach itself to your hard drive.

 

[juicedcummins]

Gary has expalined my probs here exactly. This the only site.

Not pickin on you Steve just trying to make sure that it is not brushed off as end user probs. [ DSL here]

 

[JHardwick]

Steve, first off, I can only imagine how time consuming a job like this is, and I also congratulate you on your patience. But, .....



... . all of your answers basically come back to traffic. I understand that this site has huge traffic and at times extraordinarily high peak traffic. And I also understand that the free sites don’t typically have near the traffic that this site does. I understand all of that.



But again, are you trying to haul a ton of gravel in a ½ ton truck? Apparently the free sites that load quicker have enough horsepower to do the job they need to …………. 10,000 members or a 100,000. This reminds me of the threads asking if “I can haul a 35’ 5er on my 2500”? Yeah, you can, but why struggle?



You mentioned that the $7,000 server was voted best over a couple of $14,000 servers …………. . was this bang for the buck or overall performance? We all know this game, it’s played in the truck market as well.



I’d imagine that this site takes in close to a million annually in subscription fees, plus banners which can double that figure ………. . I just don’t see any reason for the splotchy and slow loading at peak times.



-Jeff

 

[Steve St.Laurent]

They were rated #1 on overall performance. That's not to say that there weren't higher end servers out there - of course there are but these are VERY high end servers. I've tried to explain this a number of times now and I'll try once again. There have been a number of issues that have caused crashes, etc which forced reloads of software. When that happens the engine has to be retuned. To put it into a perspective that you can understand our beloved diesels can be tuned by adjusting the pressure of the fuel injected, how long it's injected, and when it's injected. What I'm talking about here is approximately 350 different variables and you cannot just max everything out - because it will perform worse and possibly just plain crash. You have to let actual traffic hit the server and then look at how that's performing and then makes some tweaks to those variables (a few at a time so you know whats working and what isn't) and see how that's performing. I'm doing that on literally a daily basis. When traffic is increasing by leaps and bounds while you are making those tweaks it complicates it. Imagine if you are working on tuning your engine for 300hp and before you get that nailed down overnight it decides to make 350hp so you work towards that and then overnight it decides to make 400hp - that's what I'm dealing with right now. The hardware IS fully up to the task it just needs to be setup right and that takes time unforetunately.



On top of that when these servers were purchased the original plan was to have one server serve up the web pages and the other to serve up the database. Once they were put into place and tuned up there was no need to do so and I then set them up in a backup situation so that if we have a server crash I can have the site back up in minutes. The traffic is now getting to the point where we are going to be at the point where we need to split that up - probably sometime later this winter. That is why I am planning a trip to Atlanta (where the servers are - I live in Michigan) so that I can make the necessary changes for that to happen. When that change is made the capacity of the system will literally be doubled.



The good news in all of this is that the servers are not crashing when the traffic volumes are hitting it in waves - there is just temporary slow performance. Back a few years ago when we went to these servers when we'd hit peak numbers we'd go into a cascade failure and everything would have to be rebuilt. Look back around the time of this post - https://www.turbodieselregister.com/forums/showthread.php?t=36710 - and you'll see what it was like then during growing pains.



No matter what we do there will always be times where the system is stressed - you simply cannot avoid it. You could go weeks with 600 simultaneous users on serving out 100 page requests per second and then you'll get all 600 requesting a page in 1 second - so the problem will never fully go away. That happens to every site out there including Microsoft, Yahoo, etc.



I do notice that your posts are always around 8:00-9:00 pm which is right smack in the middle of the busiest times of the day so you are experiencing the absolute worst of it. The times you are logging on are the times that I'm monitoring to make that nights changes to see how they work the following night - then I have to wait for that next night to see how they worked out. Some sites btw turn off certain features (like searches) during peak times to keep their basic performance up - a step which we do not wish to take. If you have the option to log in at other times - like before 8 or after 10:30 you will find very few of the problems you are experiencing at those peak times. While that is definately not an answer it may give you some temporary relief while I work on the issues.



I don't have anything to do with the financial decisions of the TDR but I can tell you that I know that your numbers of a million annually from subscriptions plus that again from banner ads is WAY WAY off. You can see the banner advertising rates under the about us menu under advertising and you'll see that there's no way the site takes in anywhere near those kind of figures. Right now there are ~15,000 subscribers on the site so you can do that math as well. The site is and has always been a free service to subscribers of the magazine. The magazine subscription is the same $35 that it has been since the magazine was founded in 1993 which is 6 years before the website came into existance. All that said finances has nothing to do with the problems - they will spend whatever is necessary to keep the performance up all I have to do is ask for the equipment. That just isn't the problem at this time. I am doing everything in my power to get the engine tuned as fast as humanly possible and I do apologize for any inconvenience you are experiencing.

 

[JHardwick]

Gotcha

 

[juicedcummins]

If we are having troubles @ peak times why can't we limit the guest users as we load up or even start dumping them? You would think members had a little priority.

 

[Steve St.Laurent]

Because if I don't see those loads I can't tune the server.

 

[JHardwick]

I wont to stir this pot anymore, because it has the ability of becoming tense.



But according to the stat sheet ... ... ...



22,667 members x $35 fee = $793,345. 00



Again, thank you for your patience.



-Jeff

 

[Steve St.Laurent]

There aren't 22,667 members. As I said above there are ~15,000 subscribers currently on the site. The other ~7,700 are people who were once subscribers and no longer are. As I also mentioned your subscription pays for the magazine and the website is a free service that comes along with that. The subscription rate is the same that it has been from the beginning and didn't change when the website came online.

 

[John - K5AWO]

No one ever said that the life of a web site administrator was supposed to be easy.



Guys, Steve is doing the best that is humanly possible. I have been in the network business over ten years and can tell you that I have never seen a more dedicated web administrator than Steve.



Steve is providing many more features on TDR than the other web sites are. The penalty for that is server overhead.



I can tell you that our company network only has about 2,000 users on it, AND it has a T3 connection or better. It gives me 5 mB transfer rates in the quiet hours. Between 0700-1000 hours, our network slows down to what a mediocre DSL line can provide at maybe 256 kbs.



Javascript is what enables the TDR website to offer the multiplicity of features and I for one don't want to see any of it crippled by turning off some of the features at times.



Tuning network variables is an intense and time consuming business that cannot be rushed. Please bear with Steve as he keeps tweaking the website for best performance. 14,400 plus users a day is intense and the queries per hour rate is almost unreal. That is approaching speeds of much higher costing systems.



Kudoes Steve for being so dedicated.



John

K5AWO

Communications since 1959.

 

[rbattelle]

I think we all agree that Steve is doing a great job. Lord knows I wouldn't want to do it (or even be capable of doing it)! Jeff just wanted to point out that we're seeing some slowness. As Steve mentions, if no one ever points these things out he might never know about them!



-Ryan

 

[Gary - K7GLD]

"I think we all agree that Steve is doing a great job. "





YUP, I seriously doubt ANY of us are suggesting otherwise - and it is as normal for us USERS to be somewhat defensive of OUR systems, as it is for Steve to be of his - after all, even DC is pretty much noted for their "they ALL do that" response to owner complaints!

 

[Steve St.Laurent]

We had 16,226 visitors to the system yesterday - an increase of almost 2,000 users in one day. This is good and bad - good that we're seeing the traffic, bad because it's driving me nuts. Based on how the traffic is increasing I'm trying to see if I can arrange to fly down to Atlanta earlier than previously planned to reconfigure the two servers - we never in a million years expected traffic to increase to these levels this fast. Traffic has increased almost 60% in the last few weeks. I'm not sure if I'm going to be able to arrange it or not as we have a lot going on through the holidays (as does pretty much everyone I'm sure). We may end up having to put in place a band aid to get us through to the beginning of January - which may be limiting the number of guests on the site or others. I am going to turn off the who's online list, the birthdays list, upcoming events, and the forum jump box at the lower right to reduce loads on the system for now. Bear with me guys.

 

[rbattelle]

Personally, I think the massive traffic increases are due to the popularity of the 3rd gen trucks and the number of people afraid of Ford. Just a theory.

 

[Ol'TrailDog]

My two cents. This is an incredibly well run site. Many nifty features and works well. Great bunch of folks and hats (cowboy that is) off to Steve for a job well done. !

 

[BigEasy]

Try dumping the POS MS IE browser and lighting up FoxFire.

http://www.mozilla.com

Thanks for the heads up on Foxfire, Dog... what a difference!

 

[Steve St.Laurent]

Well, as you may have read on another post I reconfigured the servers last night to split the web page and database functions across the two servers and thus far things are running MUCH better - we'll have to see how it goes tonight. Earlier in this thread we had a slow query rate of . 031% (~47 slow queries out of 149,024 per hour) which was on December 3rd. Right now we are running at a slow query rate of . 005% (~8. 5 slow queries out of 165,316 per hour).



Here's what our server loads are today on the web server (for the last hour):



8:20PM up 13 days, 10:11, 1 user, load averages: 0. 38, 0. 46, 0. 44

8:25PM up 13 days, 10:16, 1 user, load averages: 0. 64, 0. 45, 0. 43

8:30PM up 13 days, 10:21, 1 user, load averages: 0. 38, 0. 45, 0. 42

8:35PM up 13 days, 10:26, 1 user, load averages: 0. 51, 0. 46, 0. 42

8:40PM up 13 days, 10:31, 1 user, load averages: 0. 41, 0. 43, 0. 41

8:45PM up 13 days, 10:36, 1 user, load averages: 0. 50, 0. 55, 0. 47

8:50PM up 13 days, 10:41, 1 user, load averages: 0. 53, 0. 55, 0. 49

8:55PM up 13 days, 10:46, 1 user, load averages: 0. 38, 0. 54, 0. 50

9:00PM up 13 days, 10:51, 1 user, load averages: 0. 26, 0. 46, 0. 47

9:05PM up 13 days, 10:56, 1 user, load averages: 0. 51, 0. 45, 0. 45

9:10PM up 13 days, 11:01, 1 user, load averages: 0. 56, 0. 43, 0. 42

9:15PM up 13 days, 11:06, 1 user, load averages: 0. 35, 0. 49, 0. 46

9:20PM up 13 days, 11:11, 1 user, load averages: 0. 58, 0. 50, 0. 47



and on the database server:



8:20PM up 6 days, 20:45, 1 user, load averages: 0. 04, 0. 16, 0. 15

8:25PM up 6 days, 20:50, 1 user, load averages: 0. 26, 0. 20, 0. 17

8:30PM up 6 days, 20:55, 1 user, load averages: 0. 50, 0. 33, 0. 23

8:35PM up 6 days, 21 hrs, 1 user, load averages: 0. 11, 0. 24, 0. 22

8:40PM up 6 days, 21:05, 1 user, load averages: 0. 39, 0. 21, 0. 19

8:45PM up 6 days, 21:10, 1 user, load averages: 0. 15, 0. 24, 0. 20

8:50PM up 6 days, 21:15, 1 user, load averages: 0. 26, 0. 25, 0. 21

8:55PM up 6 days, 21:20, 1 user, load averages: 0. 34, 0. 33, 0. 26

9:00PM up 6 days, 21:25, 1 user, load averages: 0. 33, 0. 31, 0. 26

9:05PM up 6 days, 21:30, 1 user, load averages: 0. 06, 0. 16, 0. 20

9:10PM up 6 days, 21:35, 1 user, load averages: 0. 16, 0. 17, 0. 18

9:15PM up 6 days, 21:40, 1 user, load averages: 0. 48, 0. 31, 0. 22

9:20PM up 6 days, 21:45, 1 user, load averages: 0. 09, 0. 19, 0. 18

9:25PM up 6 days, 21:50, 1 user, load averages: 0. 12, 0. 13, 0. 15



Here's what they were yesterday when we were running on the one server for the same time period:



8:20PM up 12 days, 10:11, 0 users, load averages: 1. 53, 0. 92, 0. 72

8:25PM up 12 days, 10:16, 0 users, load averages: 1. 59, 1. 41, 1. 00

8:30PM up 12 days, 10:21, 0 users, load averages: 0. 95, 1. 16, 1. 00

8:35PM up 12 days, 10:26, 0 users, load averages: 1. 29, 1. 16, 1. 03

8:40PM up 12 days, 10:31, 0 users, load averages: 1. 09, 0. 99, 0. 97

8:45PM up 12 days, 10:36, 0 users, load averages: 1. 16, 1. 07, 1. 01

8:50PM up 12 days, 10:41, 0 users, load averages: 1. 19, 0. 95, 0. 95

8:55PM up 12 days, 10:46, 0 users, load averages: 0. 62, 0. 86, 0. 91

9:00PM up 12 days, 10:51, 0 users, load averages: 0. 86, 0. 84, 0. 88

9:05PM up 12 days, 10:56, 0 users, load averages: 1. 20, 0. 93, 0. 89

9:10PM up 12 days, 11:01, 0 users, load averages: 0. 53, 0. 74, 0. 80

9:15PM up 12 days, 11:06, 0 users, load averages: 1. 16, 0. 81, 0. 79

9:20PM up 12 days, 11:11, 0 users, load averages: 1. 21, 0. 93, 0. 85



1. 0 is 50% loaded and 2. 0 is 100% loaded meaning that things are going in to a queue that the server will get to when it finishes what it currently has going on. As you can see the loads are down significantly. Tonight will be the real test. Yesterday we had 16,828 visitors btw!!!!

 

[fencebuilders]

Steve,great site,kudos to you,your hard work is appreciated. That said,this site is by FAR the slowest of several i frequent each day. It sometimes causes me to avoid stopping by. It seems to me,if my memory is correct,that it started with the new "format' if that is the correct term. By the way DSL here.