jackrabbit-oak-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Mueller <muel...@adobe.com>
Subject Re: Some more benchmarks
Date Mon, 03 Jun 2013 09:51:49 GMT
Hi,

I was not talking about differences in hardware. I know using different
hardware will result in different numbers.

I was worried about results would be different if you run one test alone
versus if you run all tests. That would indicate a problem in the
benchmark (framework) itself.

But luckily, that doesn't seem to be the case. I updated the code and ran
the tests again, and now the results are different. It seems there were
changes recently that improved SmallFileWriteTest for Oak-Tar about 5
times; that's great. The result I got now are:

https://gist.github.com/anonymous/5697099


Specially the SmallFileWriteTest seems slow with Oak. The problem doesn't
seem to be actual blob handling; the profiling result shows the bottleneck
is with the (few) nodes. If I change the blob size to 0 (that is, 100
nodes with the same zero-length blob each, instead of 100 nodes with 10 KB
each), I get basically the same result, with both MongoMK and the Oak-Tar.
Maybe it's a first sign of slow "many child nodes"?

Regards,
Thomas






On 6/3/13 10:27 AM, "Jukka Zitting" <jukka.zitting@gmail.com> wrote:

>Hi,
>
>On Mon, Jun 3, 2013 at 11:09 AM, Thomas Mueller <mueller@adobe.com> wrote:
>> A bit weird is, when I run the tests separately I get different numbers:
>
>The results depend on the hardware you're using, so in general numbers
>from two different environments are not directly comparable.
>
>> In your case, the N was 304 versus 3574 (more than 10 times different),
>>in
>> my case it was 528 versus 1085 (factor 2).
>
>Even relative numbers across fixtures can be different depending on
>the varying IO/CPU/memory access costs on different environments. For
>example an SSD disk will reduce the relative advantage of the TarMK
>that "cheats" by mmapping the entire repository to memory.
>
>> How did you run the test? I will try the same command line and post my
>> results.
>
>I'm using a ec2 m1.medium instance to keep the environment stable over
>time. It would be nice to keep track of results also on different
>hardware.
>
>The command line I've used so far is simply:
>
>    $ java -jar oak-run-*.jar benchmark \
>          ReadPropertyTest SmallFileReadTest SmallFileWriteTest \
>          Jackrabbit Oak-Default Oak-Mongo Oak-Segment Oak-Tar
>
>BR,
>
>Jukka Zitting


Mime
View raw message