First, we have not upgraded the hardware recently as the hardware is plenty for our usage. We have had some hardware and software problems which has caused some issues. A few months ago our primary server crashed 3 of the 4 hard drives - which I have never ever seen before. At that point I brought our backup server online (it's backing up the database continuously and the web directories each night). Mysql needs to be tuned for our usage on each server so when there is a server change there is a period of time each time when I tune the parameters based on the usage. When the primary server was repaired the operating system had to be loaded from scratch and I had to set all of the applications back up on it. After that it was running as the slave server. Last week on wednesday night I was doing some database maintenance and rebooted the server. When it rebooted it did not come back online - I am still working on that issue. At that point I switched back to our original primary server which is whadt we are running on now. Therefore I am having to tune Mysql yet again.
Also, I have been chasing a problem that I was having with the other server with a high number of aborted clients which started happening around the same time as the software upgrade. I thought it was a software bug and have been looking for that. Since switching back to this server I've found that the problem went away. It looks like there is a problem in either the network card or the cable from that server to the public network - which we will fix now that I've narrowed it down to that.
Additionally we are getting into our peak usage time of the year which also requires me to tune Mysql to stay up. Until the traffic hits there's no way to know what needs to be tweaked which doesn't happen until evening. Then I need to wait until a slower time so that I can make the changes and reboot Mysql without kicking everybody off of the site (hence why I am up at 2am right now). These things explain some of the slow performance issues recently.
BTW, I have firewalled out pings to the server as rkressg suggested above. We have regular hack attempts on the site because of the volume of traffic we get and our high rankings on the search engines. Stopping ping traffic blocks one of their avenues for shutting down the site.
When the traffic levels get high enough then rather than having a master/slave server arrangement with one just being the backup I will split the database serving and webpage serving between the two of them. We haven't hit that point yet. This evening our server load did hit 1. 99 once (2. 0 is fully loaded) but that was only for one 5 minute period and was well below that most of the night.
Now, what we have seen is that PC issues with Javascript can cause very slow performance for individuals because this new version of vBulletin uses a lot of Javascript to operate. What internet security software are you running? Many internet security packages filter Javascript which can cause it to run very slowly. Look around in any packages that you are running and see if you can find any mention of Javascript and try disabling that security measure temporarily to see if it improves. If it does most of them will allow you to turn features off for specific sites. Also, older versions of Java can cause slowdowns - try updating your Java package and see if that improves thing. Lastly John - K5AWO found that a driver on his system was causing his slowdowns, check out this thread -
https://www.turbodieselregister.com/forums/showthread.php?t=112231 - and see if that applies to you.
I'm on the site at least 6-8 hours a day every day and usually am online during the busiest time periods and I rarely have to wait more than 10 seconds for a page to load and 98% of the time they load in 2-3 seconds. I'm on a 768k dsl connection.