Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 40183 invoked from network); 17 Apr 2011 00:03:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 17 Apr 2011 00:03:35 -0000 Received: (qmail 48785 invoked by uid 500); 17 Apr 2011 00:03:32 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 48761 invoked by uid 500); 17 Apr 2011 00:03:32 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 48753 invoked by uid 99); 17 Apr 2011 00:03:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Apr 2011 00:03:32 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a54.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Apr 2011 00:03:26 +0000 Received: from homiemail-a54.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTP id BD3B83A4058 for ; Sat, 16 Apr 2011 17:03:00 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=subject :mime-version:content-type:from:in-reply-to:date:message-id :references:to; q=dns; s=thelastpickle.com; b=FDwfLGO/8Sl0bA0Pza 6rFh2MQEQWv1qDFZDXwxnCtZ9WDfaLe6qoiCVBvcEN9KW+earLZA1OW/kPQ2gI2Z J22mUE3wtzmiZh3lh7ppgcfFk1+2Hnsgol15Y0hM5kfq9GIge0S+WcqCgOJq41zO kc4klVWh5tccxwHKMkzL8e07U= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h= subject:mime-version:content-type:from:in-reply-to:date :message-id:references:to; s=thelastpickle.com; bh=qyl8xOfFMu2qA PfgR2w5PYVmrFw=; b=MHBR9Xd6OqQCCqqNXlfQ4AzemPUh/X2i2y8k3OwDbhdWm TruBuxEn4hz/CbcGgeeXy9lkOuILH8qcagPoDNyiDZSxflLeDa9CtBF8/J0pmh71 fXGkwWwL7iXneUCUzDfWCdTu9my6H+4jFjxi13trPW16qcKGz2YTZMXep6U/Jo= Received: from [10.0.1.155] (121-73-157-230.cable.telstraclear.net [121.73.157.230]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTPSA id 0AA9E3A4057 for ; Sat, 16 Apr 2011 17:02:59 -0700 (PDT) Subject: Re: question about performance of Cassandra 0.7.4 under a read-heavy workload. Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: multipart/alternative; boundary=Apple-Mail-34-421346543 From: aaron morton X-Priority: 3 In-Reply-To: Date: Sun, 17 Apr 2011 12:02:57 +1200 Message-Id: <77CABA9F-513A-40AA-A78B-147E5092705D@thelastpickle.com> References: <75F493B1-DBD0-44E2-BCA9-296E99499CAB@thelastpickle.com> <15051806.bf1e.12f5996fe87.Coremail.sei_wjx@126.com> To: user@cassandra.apache.org X-Mailer: Apple Mail (2.1084) --Apple-Mail-34-421346543 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Am assuming you are not getting a hot spot in the ring using the OPP.=20 When running 0.7.4 with reduced concurrent reads if you see the read = stage backing up using nodetool tpstats and the output from iostats = shows that the IO system is not stressed then you should return the = concurrent reads to the recommended value.=20 Not sure why the recommendation changed, but how does 0.7.4 perform with = the recommended number of concurrent readers? Out of interest can anyone talk about IO changes in 0.7.X that resulted = in the new recommendation? Thanks Aaron On 16 Apr 2011, at 13:33, =E9=AD=8F=E9=87=91=E4=BB=99 wrote: > To make a comparation, 10 threads were run against the two workloads = seperately. below is the result of Cassandra0.7.4. > write heavy workload(i.e., write/read: 50%/50%) > median throughput: 5816 operations/second(i.e., 2908 writes and 2908 = reads) update latency:1.32ms read latency:1.81ms > read heavy workload(i.e., write/read: 5%/95%) > median throughput: 40 operations/second(i.e., 2 writes and 38 reads) = update latency:1.85ms read latency:90.43ms >=20 > and for cassandra0.6.6, the result is: > write heavy workload(i.e., write/read: 50%/50%) > median throughput: 3284 operations/second(i.e., 1642 writes and 1642 = reads) update latency:2.29ms read latency:3.51ms > read heavy workload(i.e., write/read: 5%/95%) > median throughput: 2759 operations/second(i.e., 2621 writes and 138 = reads) update latency:2.33ms read latency:3.53ms >=20 > all the tests were run in one environment. and most configurations of = cassandra are just as default, except that:we choose = orderPreservingPartitioner for all the tests and set concurrent_reads as = 8( which is the default value of 0.6.6 but the default value of 0.7.4 is = 32) .=20 >=20 >=20 >=20 >=20 > At 2011-04-16 06:53:01=EF=BC=8C"Aaron Morton" = wrote: > Will need to know more about the number of requests, iostats etc. = There is no reason for it to run slower. >=20 > Aaron > On 16/04/2011, at 2:35 AM, =E9=AD=8F=E9=87=91=E4=BB=99 = wrote: >=20 >> I just deployed cassandra 0.7.4 as a 6-server cluster and tested its = performance via YCSB. >> The result seems confusing when compared to that of Cassandra0.6.6. = Under a write heavy workload(i.e., write/read: 50%/50%), Cassandra0.7.4 = obtains a really satisfactory latency. I mean both the read latency and = write latency is much lower than those of Cassandra0.6.6. >> However, under a read heavy workload(i.e., write/read:5%/95%), = Cassandra0.7.4 performs far worse than Cassandra0.6.6 does. >>=20 >> Did I miss something? >>=20 >>=20 >> = =E4=BD=93=E9=AA=8C=E7=BD=91=E6=98=93=E9=82=AE=E7=AE=B12G=E8=B6=85=E5=A4=A7= =E9=99=84=E4=BB=B6=EF=BC=8C=E8=BD=BB=E6=9D=BE=E5=8F=91=E4=BC=98=E8=B4=A8=E5= =A4=A7=E7=94=B5=E5=BD=B1=E3=80=81=E5=A4=A7=E7=85=A7=E7=89=87=EF=BC=8C=E6=8F= =90=E9=80=9F3=E5=80=8D! >=20 >=20 > = =E4=BD=93=E9=AA=8C=E7=BD=91=E6=98=93=E9=82=AE=E7=AE=B12G=E8=B6=85=E5=A4=A7= =E9=99=84=E4=BB=B6=EF=BC=8C=E8=BD=BB=E6=9D=BE=E5=8F=91=E4=BC=98=E8=B4=A8=E5= =A4=A7=E7=94=B5=E5=BD=B1=E3=80=81=E5=A4=A7=E7=85=A7=E7=89=87=EF=BC=8C=E6=8F= =90=E9=80=9F3=E5=80=8D! --Apple-Mail-34-421346543 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Am = assuming you are not getting a hot spot in the ring using the = OPP. 

When running 0.7.4 with reduced concurrent = reads if you see the read stage backing up using nodetool tpstats and = the output from iostats shows that the IO system is not stressed then = you should return the concurrent reads to the recommended = value. 

Not sure why the recommendation = changed, but how does 0.7.4 perform with the recommended number of = concurrent readers?

Out of interest can anyone = talk about IO changes in 0.7.X that resulted in the new = recommendation?

Thanks
Aaron
=

On 16 Apr 2011, at 13:33, =E9=AD=8F=E9=87=91=E4= =BB=99 wrote:

To make a comparation, 10 threads were run against the two = workloads seperately. below is the result of Cassandra0.7.4.
write = heavy workload(i.e., write/read: 50%/50%)
median throughput:  5816 operations/second(i.e., 2908 writes and = 2908 reads) update latency:1.32ms read latency:1.81ms
read heavy = workload(i.e., write/read: 5%/95%)
median throughput:  40 = operations/second(i.e., 2 writes and 38 reads) update latency:1.85ms = read latency:90.43ms

and for cassandra0.6.6, the result = is:
write heavy workload(i.e., write/read: 50%/50%)
median throughput:  3284 operations/second(i.e., 1642 writes and = 1642 reads) update latency:2.29ms read latency:3.51ms
read heavy workload(i.e., write/read: 5%/95%)
median throughput:  2759 operations/second(i.e., 2621 writes and = 138 reads) update latency:2.33ms read latency:3.53ms

all the = tests were run in one environment. and most configurations of cassandra = are just as default, except that:we choose  = orderPreservingPartitioner for all the tests and set concurrent_reads as = 8( which is the default value of 0.6.6 but the default value of 0.7.4 is = 32) .




At 2011-04-16 = 06:53:01=EF=BC=8C"Aaron Morton" <aaron@thelastpickle.com> = wrote:
Will need to know more about the number of requests, iostats = etc. There is no reason for it to run = slower.

Aaron
On 16/04/2011, at 2:35 AM, = =E9=AD=8F=E9=87=91=E4=BB=99 <sei_wjx@126.com> = wrote:

I just = deployed cassandra 0.7.4 as a 6-server cluster and tested its = performance via YCSB.
The result seems confusing when compared to = that of Cassandra0.6.6. Under a write heavy workload(i.e., write/read: = 50%/50%), Cassandra0.7.4 obtains a really satisfactory latency. I mean = both the read latency and write latency is much lower than those of = Cassandra0.6.6.
However, under a read heavy workload(i.e., = write/read:5%/95%), Cassandra0.7.4 performs far worse than = Cassandra0.6.6 does.

Did I miss something?
=20

=E4=BD=93=E9=AA=8C=E7=BD=91=E6=98=93=E9=82=AE=E7=AE=B12G= =E8=B6=85=E5=A4=A7=E9=99=84=E4=BB=B6=EF=BC=8C=E8=BD=BB=E6=9D=BE=E5=8F=91=E4= =BC=98=E8=B4=A8=E5=A4=A7=E7=94=B5=E5=BD=B1=E3=80=81=E5=A4=A7=E7=85=A7=E7=89= =87=EF=BC=8C=E6=8F=90=E9=80=9F3=E5=80=8D!


=E4=BD=93=E9=AA=8C=E7=BD=91=E6=98=93=E9=82=AE=E7=AE=B12G= =E8=B6=85=E5=A4=A7=E9=99=84=E4=BB=B6=EF=BC=8C=E8=BD=BB=E6=9D=BE=E5=8F=91=E4= =BC=98=E8=B4=A8=E5=A4=A7=E7=94=B5=E5=BD=B1=E3=80=81=E5=A4=A7=E7=85=A7=E7=89= =87=EF=BC=8C=E6=8F=90=E9=80=9F3=E5=80=8D!

= --Apple-Mail-34-421346543--