A few notes:

* +1 for missing RF and CL cassandra stats.
* Using stripped EBS for m1.xlarge is a bad choice, unless they are using provisioned IOPS. Which they do not say. 
* Cassandra JVM settings are *not* standard. It's a low new heap size and a larger than default heap size. 
* "memtable size" which I assume they mean memtable_total_space_in_mb should default to 1/3 the heap. They have doubled it. 
* I would expect the above non standard memory settings to result in increased GC activity and increased latency / reduced throughput

* They presented the facts and said "you decide who is a winner" LOLS

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 2/10/2012, at 4:48 AM, horschi <horschi@gmail.com> wrote:

Hi Andy,

things I find odd:

- Replicacount=1 for mongo and couchdb. How is that a realistic benchmark? I always want at least 2 replicas for my data. Maybe thats just me.
- On the Mongo Config slide they said they disabled journaling. Why do you disable all safety mechanisms that you would want in a production environment? Maybe they should have added /dev/null to their benchmark ;-)
- I dont see the replicacount for Cassandra in the slides. Also CL is not specified. Imho the important stuff is missing in the cassandra configuration.
- In the goals section it said "more data than RAM". But they only have 12GB data per node, with 15GB of RAM per node!

I am very interested in a recent cassandra-benchmark, but I find this benchmark very disappointing.

cheers,
Christian


On Mon, Oct 1, 2012 at 5:05 PM, Andy Cobley <acobley@computing.dundee.ac.uk> wrote:
There are some interesting results in the benchmarks below:

http://www.slideshare.net/renatko/couchbase-performance-benchmarking

Without starting a flame war etc, I'm interested if these results should
be considered "Fair and Balanced" or if the methodology is flawed in some
way ? (for instance is the use of Amazon EC2 sensible for Cassandra
deployment) ?

Andy



The University of Dundee is a Scottish Registered Charity, No. SC015096.