Tuesday, September 29, 2009

Catalyst 5.71 vs 5.8 performance test

After a discussion on the merits of Catalyst 5.7 vs 5.8 as far as performance use goes, I decided to knock up a proper test.

I have two identical virtual machines, only on one I installed Catalyst::Runtime 5.71001 and the other with 5.80013. (Plus dependencies of course)

Running the exact same app, I hit them up with Siege for a while, results follow at the end of this email.

If you want to replicate the test or examine my extremely-simple test app, see:
Catalyst performance test app on Github (Patches gleefully accepted ;)

It's interesting to note the headline figures have 5.71 performing 316 tps, vs 5.80 making only 283 tps.
Memory usage (for this small app) has increased by 4MB, but is presumably shared. I guess I should look into that more.

The same system can serve small static pages from the webserver at about 1900 tps. A real-world application there on Cat 5.8 gets 90 tps.

I don't see that performance difference (5.71 vs 5.80) as significant, since most of your time ends up being spent in application code, rather than the Catalyst framework itself.
ie. If you want to make your code go faster, look at optimising your templating and database queries before you worry about downgrading Catalyst.

-Toby

------------------= results =----------------------
Running 10 second warmup on 5.7..
Running main test on 5.7..

Transactions: 94796 hits
Availability: 100.00 %
Elapsed time: 300.00 secs
Data transferred: 77.35 MB
Response time: 0.03 secs
Transaction rate: 315.99 trans/sec
Throughput: 0.26 MB/sec
Concurrency: 10.00
Successful transactions: 94796
Failed transactions: 0
Longest transaction: 0.98
Shortest transaction: 0.00

Process size:
101m VIRT, 34m RES



Running 10 second warmup on 5.8..
Running main test on 5.8..

Transactions: 84805 hits
Availability: 100.00 %
Elapsed time: 300.00 secs
Data transferred: 69.20 MB
Response time: 0.04 secs
Transaction rate: 282.68 trans/sec
Throughput: 0.23 MB/sec
Concurrency: 9.99
Successful transactions: 84805
Failed transactions: 0
Longest transaction: 1.07
Shortest transaction: 0.00

Process size:
103m VIRT, 38m RES

3 comments:

  1. Here are some new analysis of memory usage on my test app:

    Version: 5.7 5.8
    Rss: 35260 38768
    Private: 8244 9620
    Shared: 27016 29148


    So 5.8 is 3500kB larger, but 2000kB is shared. So we're only looking at 1500Kb more per process.

    ReplyDelete
  2. Is this on Perl 5.8 or 5.10?

    ReplyDelete
  3. Ah, those tests were on Perl 5.10.0.

    I re-ran some with Perl 5.8.8, but on a non-identical system so they weren't directly comparable. However Catalyst 5.80 on Perl 5.8.8 was about 30-40% slower than on Perl 5.10!

    ReplyDelete