Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of ryan@twitter.com designates
 209.85.160.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <AANLkTikrRSXEeyv31b0ybQnTvRxAgbFbRwERN_vK9HOk@mail.gmail.com>
References: <4780D37D-47DE-4A28-8DE7-59B119555537@clearspring.com>
	<AANLkTil46yErkh4s677tL82TsAWTXSyjBrI329BPAF2r@mail.gmail.com>
	<B97F130B-7DC8-4DB7-83DF-6994C9D17DED@clearspring.com>
	<AANLkTimmzT5eQC_G2z0gsPaesUfz_GvtP_JVK5bFhFqc@mail.gmail.com>
	<C67534AD-2BC8-4968-940A-BABFBA9AC15A@clearspring.com>
	<1279554146.314124203@192.168.2.227>
	<DDE6EA5B-3E3D-4CCB-8DE0-4808B5C7C745@gmail.com>
	<1279560166.856519322@192.168.2.227>
	<AANLkTim6tY-OYRGNc15QfyKhPfs988Kub-ZCskaJSj4G@mail.gmail.com>
	<AANLkTimWfED6fadWqwjKmGA3n4VrzWk2Apc187XoZ-vw@mail.gmail.com>
	<1376EF21-E210-4675-9518-62F4CC695F7D@gmail.com>
	<1279563944.174322970@192.168.2.227>
	<45DBBF46-1473-467C-8891-043B031095C7@gmail.com>
	<AANLkTikMP9BQuVM4h3aLYibqIR2UgQVjkN--GkImTDep@mail.gmail.com>
	<464CC098-1B5B-4E20-BFEE-DFBDA3FC6451@gmail.com>
	<AANLkTim2NcCbHk_2om-L3ofRh009qEpY_3jyAllsUV_o@mail.gmail.com>
	<AANLkTimD7Vfl0HLMg6YaNankJeA4rw480k-WxEJ8U0r8@mail.gmail.com>
	<AANLkTikrRSXEeyv31b0ybQnTvRxAgbFbRwERN_vK9HOk@mail.gmail.com>
Date: Tue, 20 Jul 2010 09:15:50 -0700
Message-ID: <AANLkTimrilpnA1Oexf0POgVgkgECQlG6D1JSnx3GFXJj@mail.gmail.com>
Subject: Re: Cassandra benchmarking on Rackspace Cloud
From: Ryan King <ryan@twitter.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Jul 20, 2010 at 6:20 AM, Juho M=E4kinen <juho.makinen@gmail.com> wr=
ote:
> I managed to run a few benchmarks.
>
> Servers =A0 r/s
> =A0 1 =A0 =A0 =A0 =A064.5k
> =A0 2 =A0 =A0 =A0 =A059.5k
>
> The configuration:
> Client: Machine with four Quad Core Intel Xeon CPU E5520 @ 2.27Ghz
> cpus (total 16 cores), 4530 bogomips per core. 12 GB ECC corrected
> memory. Supermicro mainboard (not sure about exact type).
>
> Cassandra Servers:
> =A0CPU: Intel Xeon X3450 Quad Core (total 4 cores).
> =A0Memory: 16 GB DDR SDRAM DIMM 1333 Mhz ECC memory (4 x 4GB DIMMs).
> =A0Disks: Three 1TB Samsung Spinpoint F3 HD103SJ 7300 rpm disks with
> 32MB internal cache in following configuration: single disk as root
> partition which hosts commitlog and two disks in Linux software RAID-0
> which hosts the data.
>
> - All servers are connected to one switch which has dual 1Gbps trunk
> to another switch which hosts the client. Network utilization was
> monitored during testing and it wasn't even close to saturate any
> network component.
> - All machines are using SUN JDK 1.6.0 Update 21 under RedHat
> Enterprise Linux Server release 5.3 with latests updates. Python
> version is 2.6.3.
>
> During first test with one client and one server:
> =A0- Test took 1163 seconds, resulting 64 488 reads per second.
> =A0- Cassandra server RF is set to one (RF=3D1)
> =A0- cassandra java used about 400% CPU resources which resulted with
> averate load of 8.0:
> =A0- 24% user, 20% system, 50% idle and ~6.3% software interrupts
> =A0- I used the David's modified stress.py from
> http://gist.github.com/481966 with args: ./stress.py -o read -t 50 -d
> $NODELIST -n 75000000 -k -i 2
> =A0- Java was started with following parameters: -XX:+UseCompressedOops
> -Xmx2046m -XX:TargetSurvivorRatio=3D90 -XX:+AggressiveOpts
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>
> I added another server to the cluster and tried to ran the client but
> using just the first server. This way I made sure that the server will
> redirect requests to the other server when the key hashes itself into
> it, thus making sure that cassandra works as it should. This decreased
> the performance to about 52.6kr/s which was expected, as some of the
> requests make one hop more before they can be answered.
>
> Third test was with stress.py hammering both nodes.
>
> While I was running the third test I started to thinking. With two
> nodes in a perfectly balanced ring with RF=3D1 the incoming read request
> has 50% change to hit the local server and 50% change to hit the
> another server, causing an extra network hop. When having two servers
> in the ring and client hitting only one, will be just as fast as
> having two servers and client hitting both of them. What happends when
> we add 3rd server to the ring and balance? the same, as far as I
> understant. Each incoming read request from the client has just 33%
> change to hit the local server and 66% that it needs an extra hop
> (still assuming that RF=3D1), so this doesn't make the cluster any
> faster. If my thinking is correct, we can keep adding servers and it
> will just make the cluster a bit slower because the request has less
> change to be served on the server it arrived. So in this case the
> answer to get more speed is to increase RF.
>
> But what's then the point with adding nodes into the ring? Disk speed!
> This test does reads to keys which doesn't exists, thus simulating a
> case where all requests hit the memory cache (cassandra or OS disk
> cache). Increasing node count will add more disks, so the requests are
> distributed into larger array of disks thus increasing the overall
> available IOPS throughput. So if all this was correct, we should stop
> making benchmarks which doesn't hit the disks, because it doesn't
> represent the real work scenario.

Reading non-existing keys in cassandra is cheap because they can
almost always be satisfied by a bloom filter, so you're not testing
what you think you are.

Also, you need to be very careful when using the word 'faster' when
talking about benchmarks. 'Fast' can mean one of at least two things:
latency or throughput. Make sure you know which you're measuring.

-ryan

> I didn't have time to run the test with three servers, but I'll do it
> later anyway to see what kind of results it will produce. Also doing
> the test with RF=3D2 should confirm that we can increase the cluster
> throughput by increasing the RF count even if the requests don't hit
> the disks.
>
> =A0- Juho M=E4kinen
>
>
> On Tue, Jul 20, 2010 at 12:58 AM, Dave Viner <daveviner@pobox.com> wrote:
>> I've put up a bunch of steps to get Cassandra installed on an EC2 instan=
ce:
>> http://wiki.apache.org/cassandra/CloudConfig
>> Look at the "step-by-step guide".
>> I haven't AMI-ed the result, since the steps are fairly quick and it wou=
ld
>> be just one more thing to update with a new release of Cassandra...
>> Dave Viner
>>
>> On Mon, Jul 19, 2010 at 2:35 PM, Peter Schuller
>> <peter.schuller@infidyne.com> wrote:
>>>
>>> > CPU was approximately equal across the cluster; it was around 50%.
>>> >
>>> > stress.py generates keys randomly or using a gaussian distribution, b=
oth
>>> > methods showed the same results.
>>> >
>>> > Finally, we're using a random partitioner, so Cassandra will hash the
>>> > keys using md5 to map it to a position on the ring.
>>>
>>> Ok, weird. FWIW I'll try to re-produce too on EC2 (I don't really have
>>> an opportunity to test this on real hardware atm), but no promises on
>>> when since I haven't prepared cassandra boostrapping for EC2.
>>>
>>> --
>>> / Peter Schuller
>>
>>
>