Site traffic went way up - temporary slowdowns at peak times

[Steve St.Laurent]

Our site traffic has gone way up and faster than I expected. Our RAM usage tonight went 40% over what it ever has before and that is causing the server to swap memory to disk during peak usage times - which slows things way down. We will be ordering RAM and getting it installed in the servers in the next couple of days. I made some settings changes tonight that will free up more memory but will slightly slow things down during normal times to get us through. I've also turned off the who's online and birthday lists to reduce the load. It should only be a problem around the peak times of 8:30-10:30 eastern time. Please bear with us until it's installed.



-Steve St. Laurent

Webmaster

 

[David Oxland]

Keep up the good work thanks!

 

[Sageair]

Dittos

 

[jcarew]

Thanks for doing a nice job on this site, it is appreciated

 

[Chipstien]

I see the users active is back on-line. Does this mean the ram is installed?

 

[Steve St.Laurent]

Not yet, but I've put some interim measures into place and I want to test them so I've turned all the features back on for now.

 

[Chipstien]

Everything seems normal EXCEPT when replying to a message. Takes about 3 minutes before it does something.

 

[Steve St.Laurent]

The primary server crashed earlier tonight and we are running on the slave server right now. I am shutting down the site in a few minutes to move things back over. The ram should be installed on tuesday. We had to special order the style of ram that we are looking for and that's the earliest we can work out to get it installed. I'm doing a number of interim things to try to get us through until then. Please bear with us.

 

[Gordon Maney]

I am curious, what are your RAM and typical bandwidth requirements for these forums? You do a great job with an incredibly complex set of circumstances. We salute you.

 

[Steve St.Laurent]

Here are the server specs:



Rack mount case with extra fans

Dual AMD Athlon 1900+ MP processors

Tyan S2462 dual processor motherboard

1 GB ECC RAM (error correcting)

4 - 18. 2G 15,000 RPM Ultra 160 SCSI HD's (full RAID 5 array with 1 standby HD) - hot swappable

128MB Caching RAID 5 SCSI Controller

Dual hot swappable power supplies

Dual network cards

FreeBSD



We have 2 of those. To give you an idea of the amount of heat these throw off there are 20 fans in each machine - 4 in the case, 2 in each power supply, 2 cooling each CPU, 2 cooling each hard drive. When we were in the previous co-location facility our two servers raised the temperature in the data center by 10 degrees and they had to install an additional air conditioning unit to keep up. One is the primary server that runs the site regularly. The 2nd is a slave server that continuously backs up the MySQL database and backs up the web directories every morning at 5:30. The primary server has crashed a couple times in the last couple weeks now and I've been able to switch over to the slave server in a couple of minutes with no loss of data. We originally had the machines configured with 2gb but there was a bug with the particular style of ram that they gave us that caused problems - which didn't happen with 1gb. That was in May of 2002 and our traffic has grown 235% since then. We are upping the ram to 2gb in both of the servers.



As far as bandwidth goes last month we transfered 221GB of data across the net. We are headed for 260GB this month. That is only what users request on the site. It doesn't include any of the communications between the servers (that is handled on a private network between the two) or any file transfers that I've done to and from the server. We are regularly hitting peaks of 1. 95mb/second. Our connection was throttled at 2mb/second and just upped that yesterday to 3mb/second. Our traffic is headed for a 19% growth this month, was 13% last month, and was 6-9% for the 4 months behind that.



Our MySQL server gets hit with an average of 70 database queries per second - I don't have any peak numbers on that but if I compare the average bandwidth usage to peak and apply that to the number of queries we hit approximately 170 queries per second to the MySQL server. We had several days this week that we got more than 1000 posts in the forums in a single day - our average for the month is 843 posts/day.

 

[Sticks]

:--)



So the TDR is doing good, right?



Who knew when this site started up that it would grow into this kind of beast?

 

[Bob W]

After reading the server specs that Steve posted, I feel like a flea has a higher IQ than me. Keep up the good work, Steve.

 

[jarsong]

Originally posted by Steve St. Laurent

Here are the server specs:



Rack mount case with extra fans

Dual AMD Athlon 1900+ MP processors

Tyan S2462 dual processor motherboard

1 GB ECC RAM (error correcting)

4 - 18. 2G 15,000 RPM Ultra 160 SCSI HD's (full RAID 5 array with 1 standby HD) - hot swappable

128MB Caching RAID 5 SCSI Controller

Dual hot swappable power supplies

Dual network cards

FreeBSD



We have 2 of those. To give you an idea of the amount of heat these throw off there are 20 fans in each machine - 4 in the case, 2 in each power supply, 2 cooling each CPU, 2 cooling each hard drive. When we were in the previous co-location facility our two servers raised the temperature in the data center by 10 degrees and they had to install an additional air conditioning unit to keep up. One is the primary server that runs the site regularly. The 2nd is a slave server that continuously backs up the MySQL database and backs up the web directories every morning at 5:30. The primary server has crashed a couple times in the last couple weeks now and I've been able to switch over to the slave server in a couple of minutes with no loss of data. We originally had the machines configured with 2gb but there was a bug with the particular style of ram that they gave us that caused problems - which didn't happen with 1gb. That was in May of 2002 and our traffic has grown 235% since then. We are upping the ram to 2gb in both of the servers.



As far as bandwidth goes last month we transfered 221GB of data across the net. We are headed for 260GB this month. That is only what users request on the site. It doesn't include any of the communications between the servers (that is handled on a private network between the two) or any file transfers that I've done to and from the server. We are regularly hitting peaks of 1. 95mb/second. Our connection was throttled at 2mb/second and just upped that yesterday to 3mb/second. Our traffic is headed for a 19% growth this month, was 13% last month, and was 6-9% for the 4 months behind that.



Our MySQL server gets hit with an average of 70 database queries per second - I don't have any peak numbers on that but if I compare the average bandwidth usage to peak and apply that to the number of queries we hit approximately 170 queries per second to the MySQL server. We had several days this week that we got more than 1000 posts in the forums in a single day - our average for the month is 843 posts/day.

WOW! Talk about data management!!!! No room for error at those levels. GREAT JOB!

jarsong

 

[Mniver]

Great Job Steve, Thanks for all the hard work you do to keep us up and running.

 

[Pirate4x4]

Throw more RAM at it. I've got a much larger VB than TDR, on a single server (dual 2. 8 Xeons) and am currently @ 3 gigs of ram. Honestly for peak times I need 4. I feel your pain, I've been there many times. Keep up the good work!