JMU2004 Posted February 10, 2014 Share Posted February 10, 2014 It's slow, but better than yesterday. Link to comment Share on other sites More sharing options...
dailylurker Posted February 10, 2014 Share Posted February 10, 2014 I kept restarting my phone. Glad it's not the problem. Today is the worse day of lurking yet. Link to comment Share on other sites More sharing options...
ravensrule Posted February 10, 2014 Share Posted February 10, 2014 Getting the crappy SQL errors again. Link to comment Share on other sites More sharing options...
motsco Posted February 10, 2014 Share Posted February 10, 2014 maybe sever cant handle so many F5 Key Strikes. Link to comment Share on other sites More sharing options...
yoda Posted February 10, 2014 Share Posted February 10, 2014 Lot of SQL errors reappearing again Link to comment Share on other sites More sharing options...
bmorelights Posted February 11, 2014 Share Posted February 11, 2014 As someone else somewhat experienced in PHP/MySQL development, and who runs a large message board myself, I cannot emphasize the above enough. I'm not surprised to hear that it's not a problem with overall CPU/mem load. That's rarely the case. It's far more likely the maximum number of simultaneous SQL connections being exceeded. This is where careful examination of log files, and perhaps addition of some debugging/error-handling code to the forum software, would be helpful. It may be as simple as substantially increasing a software limit, or it may require some rewriting to eliminate pconnects. Short answer - this is definitely fixable. As I said, I'd be happy to volunteer some of my time to diagnose and/or fix this... If the logs don't show what is happening, a higher log level is in order. Link to comment Share on other sites More sharing options...
gymengineer Posted February 11, 2014 Share Posted February 11, 2014 For me, the streamlined version of the site coincided with not a single SQL error appearing for several hours late afternoon through evening. Immediately upon the switch back to the original layout, I'm starting to get the SQL errors again. Link to comment Share on other sites More sharing options...
yoda Posted February 11, 2014 Share Posted February 11, 2014 Same as what gymengineer said above Link to comment Share on other sites More sharing options...
WEATHER53 Posted February 11, 2014 Share Posted February 11, 2014 Just had to re-register although I was already signed in. Where are Sean or John Wow, they can fix this? Link to comment Share on other sites More sharing options...
nj2va Posted February 11, 2014 Share Posted February 11, 2014 Awful performance tonight. It took me an hour to get back in. Link to comment Share on other sites More sharing options...
KamuSnow Posted February 11, 2014 Share Posted February 11, 2014 Very slow for me too, the whole evening. Only a couple of sql errors, but it's like permanent slow mode now. Link to comment Share on other sites More sharing options...
KamuSnow Posted February 11, 2014 Share Posted February 11, 2014 Wow, 5 minutes after I made the previous post, it got fast - thanks! Link to comment Share on other sites More sharing options...
grhqofb5 Posted February 11, 2014 Share Posted February 11, 2014 the homepage no longer lets you view the forums. Had to find a back way in to the mid atlantic section Link to comment Share on other sites More sharing options...
stormtracker Posted February 11, 2014 Author Share Posted February 11, 2014 We have guest viewing turned off, thus you can't view the board if you aren't a member/signed in. Also, from our tech guru scott, the problem isn't hardware. He says the hardware isn't even sweating: The problem is clearly software/db related and we're going to be trying some new performance saving moves (like turning off features, hooks, etc) Database: Server load 1.12 (6 CPUs) Memory Used 60.53% (3,197,988 of 5,283,704) Swap Used 4.54% (88,944 of 1,959,920) Web: Server load 4.61 (16 CPUs) Memory Used 23.55% (3,633,172 of 15,429,888) Swap Used 0.01% (196 of 1,959,920) That's with 2631 users. Servers are bored and yawning, yet still randomly sluggish with the occasional error even in performance mode. Link to comment Share on other sites More sharing options...
bmorelights Posted February 11, 2014 Share Posted February 11, 2014 We have guest viewing turned off, thus you can't view the board if you aren't a member/signed in. Also, from our tech guru scott, the problem isn't hardware. He says the hardware isn't even sweating: The problem is clearly software/db related and we're going to be trying some new performance saving moves (like turning off features, hooks, etc) So the problem isn't hardware, but the solution is to lessen the load on the hardware? Why not just figure out what is causing the sql errors? Am I missing something? Link to comment Share on other sites More sharing options...
stormtracker Posted February 11, 2014 Author Share Posted February 11, 2014 So the problem isn't hardware, but the solution is to lessen the load on the hardware? Why not just figure out what is causing the sql errors? Am I missing something? Yes, The point is to lighten the load on the dbm/sql...software. Thats the hangup. That;s what I kinda explained. Link to comment Share on other sites More sharing options...
bmorelights Posted February 11, 2014 Share Posted February 11, 2014 Yes, The point is to lighten the load on the dbm/sql...software. Thats the hangup. That;s what I kinda explained. I get what you're saying, but given everything I've seen on the site and in the thread, the temporary degradation of features is a short-term band-aid at best. A few things stand out from those hardware stats and other things in this thread: Your servers aren't using all of their ram or cpu time. This suggests that the bottleneck is IO-related, ie: your hard drives are working as hard as they can, but can't read (or write) data fast enough to keep up with demand. If you were running on Windoze servers, I'd look at whether or not you're running an x86 version of the DBMS or a x64 one. Both servers seem to be using an amount of ram equivalent to only the 32-bit memory space. That seems obvious, but could be overlooked quite easily. I'd imagine they both run linux though. Your databases are probably exhibiting one or more common scalability or maintenance problems: index and table fragmentation, inefficient or missing indexes, infrequent analysis runs(mysql)... and more. Maintenance is key... Turn on slow query logging. That will point out which queries take the longest to run on the DB server. Look at the indexes and query execution plans of your slowest queries. Unfortunately mysql doesn't offer index statistics like oracle or SQL Server, but you can still figure out where your slow queries are doing table or index scans on large tables. What does the disk config look like? If the DB server has the hardware to support it, add more disks and set up a RAID 10 or RAID 100 configuration. Even better if you're on EC2 or some other cloud server provider you can just add storage volumes and configure striping with LVM. If all that is not enough to help, consider setting up replication for your DB server. If read performance during large load events is a problem (it probably is), a read replica can help immensely with handling the read load. Other things you could do: install memcache (or some other supported cache) on the webserver and utilize some of that free ram on your webserver to cache resources that don't change between pageviews. IPB supports this. use a CDN (cloudflare is great) to deliver static resources like images, css files, javascript, etc. Those resources get cached by browsers, but with thousands hitting the site per day that's still a thimblefull in the bucket. I understand why features get turned off, but if the problem is only highlighted by high load, and getting worse over time, eventually you'll run out of things to turn off. Hope you get things figured out... I don't know weather well enough to contribute on the weather threads, but I do know software and servers... I'll shut up now. Link to comment Share on other sites More sharing options...
stormtracker Posted February 11, 2014 Author Share Posted February 11, 2014 I get what you're saying, but given everything I've seen on the site and in the thread, the temporary degradation of features is a short-term band-aid at best. A few things stand out from those hardware stats and other things in this thread: Your servers aren't using all of their ram or cpu time. This suggests that the bottleneck is IO-related, ie: your hard drives are working as hard as they can, but can't read (or write) data fast enough to keep up with demand. If you were running on Windoze servers, I'd look at whether or not you're running an x86 version of the DBMS or a x64 one. Both servers seem to be using an amount of ram equivalent to only the 32-bit memory space. That seems obvious, but could be overlooked quite easily. I'd imagine they both run linux though. Your databases are probably exhibiting one or more common scalability or maintenance problems: index and table fragmentation, inefficient or missing indexes, infrequent analysis runs(mysql)... and more. Maintenance is key... Turn on slow query logging. That will point out which queries take the longest to run on the DB server. Look at the indexes and query execution plans of your slowest queries. Unfortunately mysql doesn't offer index statistics like oracle or SQL Server, but you can still figure out where your slow queries are doing table or index scans on large tables. What does the disk config look like? If the DB server has the hardware to support it, add more disks and set up a RAID 10 or RAID 100 configuration. Even better if you're on EC2 or some other cloud server provider you can just add storage volumes and configure striping with LVM. If all that is not enough to help, consider setting up replication for your DB server. If read performance during large load events is a problem (it probably is), a read replica can help immensely with handling the read load. Other things you could do: install memcache (or some other supported cache) on the webserver and utilize some of that free ram on your webserver to cache resources that don't change between pageviews. IPB supports this. use a CDN (cloudflare is great) to deliver static resources like images, css files, javascript, etc. Those resources get cached by browsers, but with thousands hitting the site per day that's still a thimblefull in the bucket. I understand why features get turned off, but if the problem is only highlighted by high load, and getting worse over time, eventually you'll run out of things to turn off. Hope you get things figured out... I don't know weather well enough to contribute on the weather threads, but I do know software and servers... I'll shut up now. This might be helpful. Thanks! Link to comment Share on other sites More sharing options...
Ralph Wiggum Posted February 11, 2014 Share Posted February 11, 2014 Is anyone else having issues connecting via tapatalk this am? It just hangs up at the "connecting to server" page. Using a pc I have no problem however. Please help! Link to comment Share on other sites More sharing options...
mappy Posted February 11, 2014 Share Posted February 11, 2014 Couldn't get on via tapatalk either Link to comment Share on other sites More sharing options...
Ralph Wiggum Posted February 11, 2014 Share Posted February 11, 2014 Couldn't get on via tapatalk either Glad to know its not just me. Link to comment Share on other sites More sharing options...
North Balti Zen Posted February 11, 2014 Share Posted February 11, 2014 I think they disabled tapatalk yesterday during the height of the sql problems. Off for me as well still. On my mobile, having to access the site through the browser since about 2:00 yesterday. Link to comment Share on other sites More sharing options...
mappy Posted February 11, 2014 Share Posted February 11, 2014 Randy: Whatever you guys did to limit some features was really screwy when it came to logging in and the main page. I had no way of getting to the threads themselves... had to have Andy email me the link. May want to fix that. Link to comment Share on other sites More sharing options...
snowfan Posted February 11, 2014 Share Posted February 11, 2014 Randy: Whatever you guys did to limit some features was really screwy when it came to logging in and the main page. I had no way of getting to the threads themselves... had to have Andy email me the link. May want to fix that. Had the same issue and was able to get in via 'Topics of Interest' on the main page. Link to comment Share on other sites More sharing options...
mappy Posted February 11, 2014 Share Posted February 11, 2014 Had the same issue and was able to get in via 'Topics of Interest' on the main page. That's what Andy told me to do, however there was nothing listed under Topics of Interest for me. Link to comment Share on other sites More sharing options...
H2O Posted February 11, 2014 Share Posted February 11, 2014 Had the same issue and was able to get in via 'Topics of Interest' on the main page. If it wasn't for that I would still be in the Amwx store trying to buy maps from last nights euro run Link to comment Share on other sites More sharing options...
ravensrule Posted February 11, 2014 Share Posted February 11, 2014 Had the same issue and was able to get in via 'Topics of Interest' on the main page. Yup had to do the same thing since yesterday evening. Link to comment Share on other sites More sharing options...
stormtracker Posted February 11, 2014 Author Share Posted February 11, 2014 Or you guys can check your browser history. Yes, its cumbersome, but we are doing what we can to cut down on the sql errors and db resources. Looking into some more options today. sorry yall. Link to comment Share on other sites More sharing options...
ravensrule Posted February 11, 2014 Share Posted February 11, 2014 Or you guys can check your browser history. Yes, its cumbersome, but we are doing what we can to cut down on the sql errors and db resources. Looking into some more options today. sorry yall. I clear mine so my wife cannot see the naughty sites i go to . Link to comment Share on other sites More sharing options...
stormtracker Posted February 11, 2014 Author Share Posted February 11, 2014 So far so good...model run times are when we usually get SQL errors...cross your fingers. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.