Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
Sender: scode@scode.org
In-Reply-To: <AANLkTikrRSXEeyv31b0ybQnTvRxAgbFbRwERN_vK9HOk@mail.gmail.com>
References: <4780D37D-47DE-4A28-8DE7-59B119555537@clearspring.com>
	<AANLkTil46yErkh4s677tL82TsAWTXSyjBrI329BPAF2r@mail.gmail.com>
	<B97F130B-7DC8-4DB7-83DF-6994C9D17DED@clearspring.com>
	<AANLkTimmzT5eQC_G2z0gsPaesUfz_GvtP_JVK5bFhFqc@mail.gmail.com>
	<C67534AD-2BC8-4968-940A-BABFBA9AC15A@clearspring.com>
	<1279554146.314124203@192.168.2.227>
	<DDE6EA5B-3E3D-4CCB-8DE0-4808B5C7C745@gmail.com>
	<1279560166.856519322@192.168.2.227>
	<AANLkTim6tY-OYRGNc15QfyKhPfs988Kub-ZCskaJSj4G@mail.gmail.com>
	<AANLkTimWfED6fadWqwjKmGA3n4VrzWk2Apc187XoZ-vw@mail.gmail.com>
	<1376EF21-E210-4675-9518-62F4CC695F7D@gmail.com>
	<1279563944.174322970@192.168.2.227>
	<45DBBF46-1473-467C-8891-043B031095C7@gmail.com>
	<AANLkTikMP9BQuVM4h3aLYibqIR2UgQVjkN--GkImTDep@mail.gmail.com>
	<464CC098-1B5B-4E20-BFEE-DFBDA3FC6451@gmail.com>
	<AANLkTim2NcCbHk_2om-L3ofRh009qEpY_3jyAllsUV_o@mail.gmail.com>
	<AANLkTimD7Vfl0HLMg6YaNankJeA4rw480k-WxEJ8U0r8@mail.gmail.com>
	<AANLkTikrRSXEeyv31b0ybQnTvRxAgbFbRwERN_vK9HOk@mail.gmail.com>
Date: Tue, 20 Jul 2010 15:41:59 +0200
Message-ID: <AANLkTininoTxB4S87eN_DZcepoYikGcP8Ljqa4eL3vGy@mail.gmail.com>
Subject: Re: Cassandra benchmarking on Rackspace Cloud
From: Peter Schuller <peter.schuller@infidyne.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8

> But what's then the point with adding nodes into the ring? Disk speed!

Well, it may also be cheaper to service an RPC request than service a
full read or write, even in terms of CPU.

But: Even taking into account that requests are distributed randomly,
the cluster should still scale. You will approach the overhead of
taking the overhead of a level of RPC indirection for 100% of
requests, but it won't become worse than that. That overhead is still
going to be distributed across the entire cluster and you should still
be seeing throughput increasing as nodes are added.

That said, given that the test in this case is probably the cheapest
possible test to make, even in terms of CPU, by hitting non-existent
values, maybe the RPC overhead is simply big enough relative to this
type of request that moving from 1 to 4 nodes doesn't show an
improvement. Suppose for example that the cost of forwarding an RPC
request is comparabale to servicing a read request for a non-existent
key. Under those conditions, going from 1 to 2 nodes would not be
expected to affect throughput at all. Going from 2 to 3 should start
to see an improvement, etc. If RPC overhead is higher than servicing
the read, you'd see performance drop from 1 to 2 nodes (but should
still eventually start scaling with node count).

What seems inconsistent with this hypothesis is that in the numbers
reported by David, there is an initial drop in performance going from
1 to 2 nodes, and then it seems to flatten completely rather than
changing as more nodes are added. Other than at the point of
equilibrium between additional RPC overhead and additional capacity,
I'd expect to either see an increase or a decrease in performance with
each added node.

Additionally, in the absolute beginning of this thread, before the
move to testing non-existent keys, they were hitting the performance
'roof' even with "real" read traffic. Presuming such "real" read
traffic is more expensive to process than key misses on an empty
cluster, that is even more inconsistent with the hypothesis.

(I'm hoping to have time to run my test on EC2 tonight; will see.)

-- 
/ Peter Schuller