Important - Please Read! Major Changes Coming!

[Mitek]

Originally Posted by probelia

just out of curiousity what do you use to stress test?

That was going to be my question

[John Stone]

Since at least a couple of you are interested in seeing some standardized stress tests, I'm happy to take a few minutes and provide those for you.

Keep in mind that I have not yet done any performance tweaking. I expect to be able to improve on these results.

I'll start with 'ab', since just about everyone who has an apache server has that utility available to them. I'll also keep "top" running in another terminal and report system load averages while the tests are running.

Let's start very light: 1000 requests in batches of 10 requests at a time (I'm using the JSF front page as the test page for all these tests):

iron:~# ab -n1000 -c10 http://192.168.0.200/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.0.200 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
 
Server Software:        Apache
Server Hostname:        192.168.0.200
Server Port:            80
Document Path:          /
Document Length:        17534 bytes
Concurrency Level:      10
Time taken for tests:   0.240681 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      17722000 bytes
HTML transferred:       17534000 bytes
Requests per second:    4154.88 [#/sec] (mean)
Time per request:       2.407 [ms] (mean)
Time per request:       0.241 [ms] (mean, across all concurrent requests)
Transfer rate:          71904.31 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     1    1   1.9      1      12
Waiting:        0    1   1.7      1      11
Total:          1    1   1.9      1      12
Percentage of the requests served within a certain time (ms)
  50%      1
  66%      2
  75%      2
  80%      3
  90%      4
  95%      6
  98%      7
  99%      8
 100%     12 (longest request)

That wasn't much of a challenge. The load average didn't even move from 0.00, and all 1000 pages were served up in less than 3/10th of second each.


Let's increase the challenge exponentially: 10,000 page requests, 100 requests at a time:

iron:~# ab -n10000 -c100 http://192.168.0.200/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.0.200 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
 
Server Software:        Apache
Server Hostname:        192.168.0.200
Server Port:            80
Document Path:          /
Document Length:        17534 bytes
Concurrency Level:      100
Time taken for tests:   1.953154 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      177220000 bytes
HTML transferred:       175340000 bytes
Requests per second:    5119.92 [#/sec] (mean)
Time per request:       19.532 [ms] (mean)
Time per request:       0.195 [ms] (mean, across all concurrent requests)
Transfer rate:          88608.48 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       4
Processing:     4   18   5.7     17      64
Waiting:        1   18   5.6     17      63
Total:          5   18   5.7     17      64
Percentage of the requests served within a certain time (ms)
  50%     17
  66%     19
  75%     20
  80%     21
  90%     25
  95%     31
  98%     37
  99%     42
 100%     64 (longest request)

Still not much of a challenge. 10,000 page requests were served up in less than 2 seconds, the majority of which took 17ms each. The longest request took just 64ms to be fulfilled. The server barely noticed: the load average peaked at 0.01.


I'm going to increase the load exponentially once again: 100,000 page requests, 1,000 requests at once:

iron:~# ab -n100000 -c1000 http://192.168.0.200/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.0.200 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests
 
Server Software:        Apache
Server Hostname:        192.168.0.200
Server Port:            80
Document Path:          /
Document Length:        17534 bytes
Concurrency Level:      1000
Time taken for tests:   26.715080 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      1772200000 bytes
HTML transferred:       1753400000 bytes
Requests per second:    3743.20 [#/sec] (mean)
Time per request:       267.151 [ms] (mean)
Time per request:       0.267 [ms] (mean, across all concurrent requests)
Transfer rate:          64782.29 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   54 659.5     10   21005
Processing:     1   98 610.3     52   24253
Waiting:        0   82 609.7     33   24220
Total:          2  152 936.2     65   24267
Percentage of the requests served within a certain time (ms)
  50%     65
  66%     69
  75%     71
  80%     73
  90%     76
  95%     81
  98%    793
  99%   3063
 100%  24267 (longest request)

As you can see, this test got the server's attention! It served up 100,000 pages--1,000 requests at a time--in less than 30 seconds, but it sure had to work for it. The load average shot up to 5.98, but there were no failed requests. Even at that load, 95% of the pages were served up in less than 1/10th of a second. Also, less than 1 Gig of the available 16 Gigs of RAM were needed. Not too bad.

[gravityhomer]

Out of curiousity, what's the load average out of? what's the units?

[John Stone]

Originally Posted by gravityhomer

Out of curiousity, what's the load average out of? what's the units?

It's magic.

Seriously, that's a complicated question. It shouldn't be, but it is. Here's a summary from a much longer multi-part article that attempts to demystify "load average" under Linux:

So, what have we learned? Those three innocuous looking numbers in the LA triplet have a surprising amount of depth behind them. The triplet is intended to provide you with some kind of information about how much work has been done on the system in the recent past (1 minute), the past (5 minutes) and the distant past (15 minutes). As you will have discovered if you tried the LA Triplets quiz, there are problems:

The "load" is not the utilization but the total queue length.
They are point samples of three different time series.
They are exponentially-damped moving averages.
They are in the wrong order to represent trend information.

These inherited limitations are significant if you try to use them for capacity planning purposes. I'll have more to say about all this in the next online column Load Average Part II: Not Your Average Average.

Personally I use it to see when things are starting to go bad. I have a terminal open to my right at all times that shows the JSF servers' load average. When it's less than 3 or so, all is well. Once it starts getting above that, things start slowing. Around 15+ all hell breaks loose.

Of course the numbers I just mentioned are based on the existing servers, which are single-CPU systems. As I understand it, load averages should be divided by the number of processors on a multi-cpu system. The new servers have 2 quad-core CPUs. I'm unclear on if that means I divide by 8 or divide by 2 in order to get a reasonable approximation compared to the old servers.

[John Stone]

Just for fun and comparison, I thought I'd duplicate the same tests I just performed on the new web server on the current web server. The results are not pretty.

1,000 connections, 10 at a time:

iron:~# ab -c10 -n1000 http://www.johnstonefitness.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.johnstonefitness.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Finished 1000 requests
 
Server Software:        Apache/2.0.58
Server Hostname:        www.johnstonefitness.com
Server Port:            80
Document Path:          /
Document Length:        15450 bytes
Concurrency Level:      10
Time taken for tests:   24.884536 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      15664064 bytes
HTML transferred:       15475688 bytes
Requests per second:    40.19 [#/sec] (mean)
Time per request:       248.845 [ms] (mean)
Time per request:       24.885 [ms] (mean, across all concurrent requests)
Transfer rate:          614.68 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       36   45  95.5     41    3048
Processing:   167  201  16.6    199     369
Waiting:       40   49   9.4     47     128
Total:        213  247  97.9    240    3260
Percentage of the requests served within a certain time (ms)
  50%    240
  66%    244
  75%    247
  80%    248
  90%    255
  95%    271
  98%    332
  99%    374
 100%   3260 (longest request)

Ugh! A mere 1,000 page requests at the very low rate of 10 requests per second took almost as long as the new server took to serve 100,000 pages at the rate of 1,000 simultaneous requests. The new server didn't move from a 0.00 load average, but the existing web server peaked at a 2.96 load average. That load average does includes normal traffic (which is very light ATM), but still....

That's pretty pitiful performance for such a light request.


Here's 10,000 page requests, 100 at a time:

iron:~# ab -c100 -n10000 http://www.johnstonefitness.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.johnstonefitness.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Finished 10000 requests
 
Server Software:        Apache/2.0.58
Server Hostname:        www.johnstonefitness.com
Server Port:            80
Document Path:          /
Document Length:        15450 bytes
Concurrency Level:      100
Time taken for tests:   215.113645 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      156566792 bytes
HTML transferred:       154683596 bytes
Requests per second:    46.49 [#/sec] (mean)
Time per request:       2151.136 [ms] (mean)
Time per request:       21.511 [ms] (mean, across all concurrent requests)
Transfer rate:          710.77 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       39  511 739.5    377    9425
Processing:   375 1629 1023.4   1489   16971
Waiting:      161  551 842.5    398   16186
Total:        417 2140 1270.9   1875   19049
Percentage of the requests served within a certain time (ms)
  50%   1875
  66%   1935
  75%   1985
  80%   2037
  90%   3185
  95%   4599
  98%   5914
  99%   7857
 100%  19049 (longest request)

The new server performed the above test in less than two seconds. The existing server? It took a whopping 3 minutes 35 seconds! That's like four quadrillion percent longer. Also, on the existing server 95% of the requests took up tp 5 full seconds to complete. The new server? 95% of the request were completed in less than 3/100th of a second.


I probably shouldn't even run this final test on the existing server, but here it goes: 100,000 page requests at the rate of 1,000 requests at once...

iron:~# ab -c1000 -n100000 http://www.johnstonefitness.com/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/
Benchmarking www.johnstonefitness.com (be patient)
apr_socket_recv: Connection timed out (110)
Total of 8086 requests completed

Ha, the server stopped responding before it could even complete 10% of the test.

Yeah, things are going to be much, much faster around here real soon now. It's not an exaggeration to say the new web server is literally 100 times faster than the old web server.

[gravityhomer]

Originally Posted by John Stone

Yeah, things are going to be much, much faster around here real soon now. It's not an exaggeration to say the new web server is literally 100 times faster than the old web server.

So, is it safe to say that our connections will be the limiting factor for speed?

[skydude2020]

hey john, have you bothered to look into services like Amazon's Web Services (AWS), their EC2 (cloud computing) and S3 (storage service)? It's something I've been playing with recently for some organizations I work for and they seem great in-so-far-as to avoid expensive hardware costs. As there seems to be a growing set of methods to distribute the load of heavily-used web servers/services, I'm interested in others thought on utilizing such tools.

[John Stone]

Originally Posted by skydude2020

hey john, have you bothered to look into services like Amazon's Web Services (AWS), their EC2 (cloud computing) and S3 (storage service)? It's something I've been playing with recently for some organizations I work for and they seem great in-so-far-as to avoid expensive hardware costs. As there seems to be a growing set of methods to distribute the load of heavily-used web servers/services, I'm interested in others thought on utilizing such tools.

I looked at S3 awhile back, and I also briefly looked at EC2. I was turned off by the costs of EC2. One extra large instance running 24/7 (one extra large instance is what most closely mirrors just one of our new severs) put the costs around $500.00/mo--and that's the price for computing power that only approaches the power of one of our new servers. Two instances would double those monthly costs.

Once you add in data transfer costs (which, of course, is a significant monthly expense) the numbers just didn't work out.

[John Stone]

Progress update & discussion of the new blog feature here.

[Reno_1ted]

John,

When the forum changes are ready to "go live" can you post an idiots guide / laymans term explaination of what the changes are and what it means. I am pretty computer savvy but i have been lost by many of the recent updates on the main page, and sometimes felt like i was reading a textbook! If i was lost with my IT level of knowlege, there is bound to be some people here completely flumoxed and perhaps even put off by the changes, simply because they dont understand them. Many people just know the basics of the internet, how to log on, post a reply to a forum etc. They would probably be 100% confused by the recent weeks posts.

The updates seem to assume everyone has your background and knowlege of IT. RSS feeds, pingbacks etc probably mean nothing to many people.

Is a pingback when you try and fire a rubber band at someone and it backfires?

[John Stone]

Originally Posted by Reno_1ted

John,

When the forum changes are ready to "go live" can you post an idiots guide / laymans term explaination of what the changes are and what it means. I am pretty computer savvy but i have been lost by many of the recent updates on the main page, and sometimes felt like i was reading a textbook! If i was lost with my IT level of knowlege, there is bound to be some people here completely flumoxed and perhaps even put off by the changes, simply because they dont understand them. Many people just know the basics of the internet, how to log on, post a reply to a forum etc. They would probably be 100% confused by the recent weeks posts.

The updates seem to assume everyone has your background and knowlege of IT. RSS feeds, pingbacks etc probably mean nothing to many people.

I sincerely apologize!

I've tried to keep the updates on the main page as nontechnical as possible. I attempted to save the more technical banter for the discussion thread. Unfortunately it seems that my sense of perspective is a little skewed in this area. Again, I'm very sorry if I've added to anyone's confusion.

The forum will continue to work just as it always has, so there will not be a learning curve for anyone who's not interested in the new features (or simply wants to pick up the new stuff at their own pace).

That said, I will be posting detailed information on all the new features for anyone who is interested in learning about them. New terms will be explained, and all the new software features will be completely demystified. I assure you, none of it is complicated, and it's unfortunate that I may have created the opposite impression by inadvertently peppering my updates with a little too much technical jargon. I'll try and do better.

Is a pingback when you try and fire a rubber band at someone and it backfires?

Pingbacks/trackbacks are two different forms of linkbacks. Linkbacks simply allow authors of blogs (or other web documents) to know when someone else links to their work.

More on linkbacks.

[probelia]

Excellent stuff John, the blog feature is something that I'm really looking forward to.

[Reno_1ted]

No problemo, no need to apologise.

[John Stone]

Originally Posted by probelia

Excellent stuff John, the blog feature is something that I'm really looking forward to.

I've been playing around with it on the new server and it's really, really cool!

[OnConstant]

Originally Posted by jk0

If you're anything like me you'd like Debian better than Ubuntu. Ubuntu is great for a desktop distribution but it just rubs me the wrong way for servers. They pack it full a bunch of crap that shouldn't be there in a fresh install. Debian is a great vanilla platform to start out on.

Gentoo on the other hand... I guess it's nice if you don't mind compiling everything (not that you're not already aware of that). It's caused me more headaches when a package breaks or fails to compile than anything else. Now days it's just not worth the hassle. You wont get any extra performance benefits compiling everything like you would have a few years ago.

What kind of hardware are we talking here? Have you considered virtualization? Buy one nice server with 16GB of RAM and go nuts. VMware Server and ESX (especially) make me .

EDIT: I see you just posted hardware details. With another 8GB of RAM that would make one hell of a VM host.

I couldn't agree more.. I use Ubuntu as a client OS, but never as a server OS.

[mustbesix]

Are concerned that it will become harder to enforce the rules with all of these new features like blogs?

[John Stone]

Originally Posted by mustbesix

Are concerned that it will become harder to enforce the rules with all of these new features like blogs?

Not at all. Problematic users are rare here, because they never last long.

Quite a few of the members do an extremely good job of bringing any issues to my attention, and the new features won't change that.

[gazareth]

Hi John, sounds like you are having fun with all the new kit. I just wanted to post and give my appreciation for all the hard work you do with JSF. I know that this site has definitely changed my life for the better

I also have a question. Will the new blog system be able to integrate smoothly with the current membership level system? What I'm getting at is, if I have a blog instead of a journal thread, will I still be able to have it in a "private" members-only sub-section?

[Tony Yates]

Servers look like the 2950's. They are rock solid if so. I manage a VMWare farm racked with them and they are as good as the HP DL380 and the like. Good choice.

[John Stone]

Originally Posted by gazareth

I also have a question. Will the new blog system be able to integrate smoothly with the current membership level system? What I'm getting at is, if I have a blog instead of a journal thread, will I still be able to have it in a "private" members-only sub-section?

That's a great question.

The answer, I believe (I'll let you know if I find out otherwise) is "no, but...". Because blogs are not kept in a particular section of the forum, they are open to everyone by default--however you do have come control over who can view it. Here's a screenshot of the "privacy" section of the blog control panel:

_________________________________________



_________________________________________

Further, you can choose to moderate all comments, or even stop comments from being posted altogether.

I think it would be cool to let the blog author choose which usergroups can view his/her blog, and I've asked vB about that functionality. Maybe it will happen in a future release or someone will create a hack. I'll keep you posted with any news.

Originally Posted by Tony Yates

Servers look like the 2950's. They are rock solid if so. I manage a VMWare farm racked with them and they are as good as the HP DL380 and the like. Good choice.

Yep, Poweredge 2950 IIIs

I love OMSA! I wrote a shell script so I get paged if anything at all is amiss. It's neat to open the case or pull a hot drive and have my pager go off seconds later alerting me to that fact.