Server problems

[Steve St.Laurent]

FYI - our primary web server's drive array completely crashed today. We are now running on a single server so I expect that you may/will experience slowdowns. We have hardware on the way to be put in place tonight and I will get the primary server back up and running ASAP. There shouldn't be much data loss at all. The only things lost should be any classified ads placed and pictures uploaded to the readers rigs gallery since 1 am. Sorry for any inconvenience.



-Steve St. Laurent

Webmaster

 

[GFrance]

Steve you want me to send you my network administrator? She can build servers with her eyes closed. But then she wont get home until saturday. Shes in Las Vegas for some training this week. Hope she don't gamble on that Gov. credit card.

 

[JasonCzerak]

FYI - our primary web server's drive array completely crashed today. We are now running on a single server so I expect that you may/will experience slowdowns. We have hardware on the way to be put in place tonight and I will get the primary server back up and running ASAP. There shouldn't be much data loss at all. The only things lost should be any classified ads placed and pictures uploaded to the readers rigs gallery since 1 am. Sorry for any inconvenience.



-Steve St. Laurent

Webmaster

So, how many drives did you lose at once?



Try RAID 1+0 configuration this time. it can withstand more then 2 dead drives Right now I have a system with 3 dead drives in one array. . If it wasn't for service contract politics... . Well, that's another story.

 

[formula]

Right now I have a system with 3 dead drives in one array. . If it wasn't for service contract politics... . Well, that's another story.

Livin' on the edge eh? Got backup?



Sean

 

[Steve St.Laurent]

It was a RAID 5+1 array - have no idea how many drives went. I'm assuming it's probably the controller and not the drives.

 

[mjendrejcak]

What hardware are you running?

 

[JasonCzerak]

Livin' on the edge eh? Got backup?



Sean

yep got lots of them backups.



And it's only a Test machine, I can recover 100% in a few hours.



If the right combo of drives do fail from this point forward and bring down the array, I'll just rebuild the array degraded with the remaing drives. The size of the array won't change, I'll have just 4 mirrored devices in degraded mode. I'll have to flip flop some drives around in the configuration, don't even have to touch the hardware.



This is a 20 drive array configured in a Raid 1+0 with 5 controlers. No worries. Just freaking renew the service contract like you should of and I wouldn't be cutting it close. Big companies are slow and suck.



it kinda looks like this now:



1 - 2

3 - 4 (bad)

5 - 6

7 - 8

9 - 10

11 - 12

13 - 14 ( bad)

15 - 16 ( bad)

17 - 18

19 - 20



So if 15, 13, or 3 fails, the Stipe will fail and I'm SOL. I can lose 2,6,8,10,12,18 and 20 (or the mirror counter part) and still be up.









Gota love Raid 10 tho



http://www.ofb.net/~jheiss/raid10/

 

[JasonCzerak]

It was a RAID 5+1 array - have no idea how many drives went. I'm assuming it's probably the controller and not the drives.

So you are mirroring 2 Raid 5 arrays on one controler? or is the +1 a hot spare in your case?

 

[Steve St.Laurent]

Right now we have 2 servers:



Rack mount case with extra fans

Dual AMD Athlon 1900+ MP processors

Tyan S2462 dual processor motherboard

2 GB ECC RAM (error correcting)

4 - 18. 2G 15,000 RPM Ultra 160 SCSI HD's (full RAID 5 array with 1 standby HD) - hot swappable

128MB Caching RAID 5 SCSI Controller

Dual hot swappable power supplies

Dual gigabyte network cards

FreeBSD



We've been running on these servers for 4 years now. Originally we were running on only one as we are right now with the other one just on a standby backup. Later I changed them over so that the primary server just served out the web pages and was a slave SQL server to the database server (doing a continuous backup) and the database server did just the SQL and backed up the web server every night. In the near future these two servers are going to both be database servers on the back end (to provide redundancy) with a load balancer on the front end with a many web servers as we need (will probably start with 2). The plan was and is to put those in place this summer before the fall traffic increase.

 

[Steve St.Laurent]

The +1 is a hot spare in the case. Then everything is backed up between the two servers as well.

 

[JasonCzerak]

Right now we have 2 servers:



Rack mount case with extra fans

Dual AMD Athlon 1900+ MP processors

Tyan S2462 dual processor motherboard

2 GB ECC RAM (error correcting)

4 - 18. 2G 15,000 RPM Ultra 160 SCSI HD's (full RAID 5 array with 1 standby HD) - hot swappable

128MB Caching RAID 5 SCSI Controller

Dual hot swappable power supplies

Dual gigabyte network cards

FreeBSD



We've been running on these servers for 4 years now. Originally we were running on only one as we are right now with the other one just on a standby backup. Later I changed them over so that the primary server just served out the web pages and was a slave SQL server to the database server (doing a continuous backup) and the database server did just the SQL and backed up the web server every night. In the near future these two servers are going to both be database servers on the back end (to provide redundancy) with a load balancer on the front end with a many web servers as we need (will probably start with 2). The plan was and is to put those in place this summer before the fall traffic increase.

sounds cool.



Cheap, 1U boxes. No hardware RAID, just software mirror of the OS + the application. $3,000 if that on the front end.



Rebuild the DB servers one at a time with RAID 1+0 and 2 controlers. It's the only way to do it.

 

[Gary - K7GLD]

As of right now, I see a "image server down" message attached in my lead post - and no pics displayed in this thread:



http://turbodieselregister.com/forums/showthread.php?t=161002



... But the external image host shows those pics just fine - Part of the same failure?

 

[Nate]

Sheez. you guys are speaking in 00101010101010101000011101010 to me.



I am happy I can do HTML and write emails, never mind all the tech stuff! I did build my last 2 computers, but that is cake.

 

[Bob Wagner]

seems to be running like a cummins again

 

[Steve St.Laurent]

Gary - there's something blocking that in the firewall. Our FreeBSD security guru is the guy working on the other server right now so I'll have him look at it when he gets the chance.



It looks like the other server is majorly broke. He's pulled it from the facility and is working on it now. It looks like we have a motherboard issue or something else. He wasn't able to see the hard drives at all unless he pulled the network card and even then he could see them there but couldn't read from or write to them. Looks like we may be running on the one server for a little while. Thus far the loads look great on the server but I'm going to monitor them closely the rest of the night. I'm going to be in Indy over the weekend for the IRP event and I'd rather not put a fresh server into use right before going out of town because there's sure to be problems. Hang with us. . . . .

 

[PRout]

Nice job Steve & Crew. . We appreciate it.

Seems to be working ok on this end, Just a little slower than normal loading some graphics. No biggie.

 

[Steve St.Laurent]

Gary, the linked images are working again.

 

[hsmith]

We will be patient and supportive!

 

[Cattletrkr]

Now I know what my friends feel like when I start talking trucks and their eyes gloss over...

 

[PatrickCampbell]

Haha Cattle that is me at work and I work with computers!!!!