After reading Graeme's benchmark of Rails versus Grails last night, I was troubled. Something didn't seem right, but I couldn't put my finger on it. Around 3 am...it hit me...Graeme's test was comparing a webapp deployed in Tomcat, something that's threaded out of the box, with one single mongrel process...something not threaded at all. So the comparison isn't fair (albeit unintentionally), because it compares multiple Grails threads to one mongrel thread.
So, I set out to correct it. I've used mongrel with nginx in the past, and while I'm no expert, I'm comfortable setting it up.
I've run the tests myself, leaving Tomcat un-throttled. I've listed what I 've found below. For your reference, the order of the tests is:
- list
- createTest
- queryTest
- updateTest
- updateTest2
- updateTest3
You can see how these are defined in Graeme's comparison (linked above).
With 10 mongrel processes and 25 nginx processes...
(rails on left, grails on right):
[/home/wellsj/benchmark/rails_final]-] diff output.mongrel10_nginx25 ../grails_05/output.grails | grep 'Requests per'
< Requests per second: 71.78 [#/sec] (mean)
> Requests per second: 43.87 [#/sec] (mean)
< Requests per second: 84.07 [#/sec] (mean)
> Requests per second: 110.14 [#/sec] (mean)
< Requests per second: 13.04 [#/sec] (mean)
> Requests per second: 8.10 [#/sec] (mean)
< Requests per second: 55.69 [#/sec] (mean)
> Requests per second: 57.17 [#/sec] (mean)
< Requests per second: 58.30 [#/sec] (mean)
> Requests per second: 58.62 [#/sec] (mean)
< Requests per second: 73.21 [#/sec] (mean)
> Requests per second: 159.48 [#/sec] (mean)
If I up the number of mongrel threads to 25, I get:
[/home/wellsj/benchmark/rails_final]-] diff output.mongrel25_nginx25 ../grails_05/output.grails | grep 'Requests per'
< Requests per second: 64.40 [#/sec] (mean)
> Requests per second: 43.87 [#/sec] (mean)
< Requests per second: 67.59 [#/sec] (mean)
> Requests per second: 110.14 [#/sec] (mean)
< Requests per second: 13.22 [#/sec] (mean)
> Requests per second: 8.10 [#/sec] (mean)
< Requests per second: 54.36 [#/sec] (mean)
> Requests per second: 57.17 [#/sec] (mean)
< Requests per second: 58.99 [#/sec] (mean)
> Requests per second: 58.62 [#/sec] (mean)
< Requests per second: 69.21 [#/sec] (mean)
> Requests per second: 159.48 [#/sec] (mean)
So, upping the number of mongrel threads from 10 to 25 actually decreased performance.
Taking the first as our reference, it looks like Grails beats Rails handily on the create test, narrowly on the update test and update_test2, and blows it away on the update_test3. Rails beats Grails pretty well on the list test, and the query test.
Someone, please correct me if I've made a serious error here. My understanding of Tomcat threading is vague at best, so I could be missing something significant. And yes, I tried limiting Tomcat to 25 threads by setting minSpareThreads, maxSpareThreads, and maxThreads all to 25 in server.xml. While I didn't investigate these settings in depth, one or more had the desired effect...my catalina.out log filled with thread busy errors during the create test. However, the difference in results was negligible...so performance doesn't seem to be thread-bound in this case.
I'd like to also add that Rails improved significantly by going from one mongrel to the nginx/mongrel combo with 10 mongrels in the background. Here's the comparison (1 on the left, 10 on the right):
< Requests per second: 39.50 [#/sec] (mean)
> Requests per second: 71.78 [#/sec] (mean)
< Requests per second: 48.52 [#/sec] (mean)
> Requests per second: 84.07 [#/sec] (mean)
< Requests per second: 6.58 [#/sec] (mean)
> Requests per second: 13.04 [#/sec] (mean)
< Requests per second: 32.00 [#/sec] (mean)
> Requests per second: 55.69 [#/sec] (mean)
< Requests per second: 35.20 [#/sec] (mean)
> Requests per second: 58.30 [#/sec] (mean)
< Requests per second: 46.66 [#/sec] (mean)
> Requests per second: 73.21 [#/sec] (mean)
Btw, FYR, here are my machine specs:
Ubuntu Linux (Edgy)
Dell Precision M65 (laptop)
1.83 GHZ dual core Intel processor
2GB RAM
4 comments:
John - what is NGinx doing for you here? Is it doing any caching (FastCGI etc?)? may skew resutls with list and qyery tests. i don't know, just asking
Hi Marc, no, best I can tell nginx does no caching. See for yourself: http://nginx.net
Yes I did beforehand, and saw these:
"the accelerated support without caching of the remote FastCGI servers, simple load balancing and fault tolerance;
the modular architecture, the filters including gzipping, byte ranges, chunked responses, and SSI-filter; the several subrequests in one page handling in SSI-filter via FastCGI or proxy are running in parallel;"
I just wondered if any of these would have applied to your test scenario, just so we can be sure we're getting as close to like for like as possible
I did some further testing myself and setup Rails with Pound and 10 mongrel processes and Grails with only 10 threads and found the opposite
Rails degraded in performance and Grails improved in performance (significantly)
See updated benchmarks: http://docs.codehaus.org/display/GRAILS/Grails+vs+Rails+Benchmark
I'll have a look at nginx to see if it improves Rails and maybe compared with 25 mongrels and 25 Grails threads to see that performance comparison
Post a Comment