Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0DE4D9EE5 for ; Wed, 30 May 2012 18:44:13 +0000 (UTC) Received: (qmail 63658 invoked by uid 500); 30 May 2012 18:44:10 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 63636 invoked by uid 500); 30 May 2012 18:44:10 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 63628 invoked by uid 99); 30 May 2012 18:44:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 May 2012 18:44:10 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a80.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 May 2012 18:44:05 +0000 Received: from homiemail-a80.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a80.g.dreamhost.com (Postfix) with ESMTP id 4CF7B37A07A for ; Wed, 30 May 2012 11:43:43 -0700 (PDT) Received: from [172.16.1.4] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a80.g.dreamhost.com (Postfix) with ESMTPSA id 4417537A073 for ; Wed, 30 May 2012 11:43:42 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/alternative; boundary="Apple-Mail=_5D32A979-1EE9-4E79-83F1-0DD33B070EE5" Subject: Re: Replication factor Date: Thu, 31 May 2012 06:43:40 +1200 In-Reply-To: <2C85E14562B39345BCCAD90B8E7955C9149A2B@DKEXC002.adform.com> To: user@cassandra.apache.org References: <5465E653-8578-42EF-A849-360610DC11DB@thelastpickle.com> <2C85E14562B39345BCCAD90B8E7955C913E4B4@DKEXC002.adform.com> <2C85E14562B39345BCCAD90B8E7955C914692F@DKEXC002.adform.com> <59E65970-9699-4517-BF7D-3E1B6E8007A4@thelastpickle.com> <2C85E14562B39345BCCAD90B8E7955C9149A2B@DKEXC002.adform.com> Message-Id: <4296C4C0-B683-41F0-9327-0CB41731C50D@thelastpickle.com> X-Mailer: Apple Mail (2.1278) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_5D32A979-1EE9-4E79-83F1-0DD33B070EE5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Ah. The lack of page cache hits after compaction makes sense. But I = don't think the drastic effect it appears to have is expected. Do you = have an idea of how much slower local reads get ? If you are selecting coordinators based on token ranges the DS is not as = much. It still has some utility as the Digest reads will be happening on = other nodes and it should help with selecting them.=20 Thanks for the extra info.=20 Aaron ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 25/05/2012, at 1:24 AM, Viktor Jevdokimov wrote: > All data is in the page cache. No repairs. Compactions not hitting = disk for read. CPU <50%. ParNew GC <100 ms in average. > =20 > After one compaction completes, new sstable is not in page cache, = there may be a disk usage spike before data is cached, so local reads = gets slower for a moment, comparing with other nodes. Redirecting almost = all requests to other nodes finally ends up with a huge latency spike = almost on all nodes, especially when ParNew GC may spike on one node = (>200ms). We call it =93cluster hiccup=94, when incoming and outgoing = network traffic drops for a moment. > =20 > And such hiccups happens several times an hour, few seconds long. = Playing with badness threshold did not gave a lot better results, but = turning DS off completely fixed all problems with latencies, node = spikes, cluster hiccups and network traffic drops. > =20 > In our case, our client is selecting endpoints for a key by = calculating a token, so we always hit a replica. > =20 > =20 >=20 >=20 > Best regards / Pagarbiai > Viktor Jevdokimov > Senior Developer >=20 > Email: Viktor.Jevdokimov@adform.com > Phone: +370 5 212 3063, Fax +370 5 261 0453 > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania > Follow us on Twitter: @adforminsider > What is Adform: watch this short video > >=20 > Disclaimer: The information contained in this message and attachments = is intended solely for the attention and use of the named addressee and = may be confidential. If you are not the intended recipient, you are = reminded that the information remains the property of the sender. You = must not use, disclose, distribute, copy, print or rely on this e-mail. = If you have received this message in error, please contact the sender = immediately and irrevocably delete this message and any copies. >=20 > From: aaron morton [mailto:aaron@thelastpickle.com]=20 > Sent: Thursday, May 24, 2012 13:00 > To: user@cassandra.apache.org > Subject: Re: Replication factor > =20 > Your experience is when using CL ONE the Dynamic Snitch is moving = local reads off to other nodes and this is causing spikes in read = latency ?=20 > =20 > Did you notice what was happening on the node for the DS to think it = was so slow ? Was compaction or repair going on ?=20 > =20 > Have you played with the badness threshold = https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L472 = ?=20 > =20 > Cheers > =20 > =20 > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > =20 > On 24/05/2012, at 5:28 PM, Viktor Jevdokimov wrote: >=20 >=20 > Depends on use case. For ours we have another experience and = statistics, when turning dynamic snitch off makes overall latency and = spikes much, much lower. > =20 > =20 > =20 >=20 > Best regards / Pagarbiai > Viktor Jevdokimov > Senior Developer > =20 > Email: Viktor.Jevdokimov@adform.com > Phone: +370 5 212 3063, Fax +370 5 261 0453 > J. Jasinskio 16C, LT-01112 Vilnius, Lithuania > Follow us on Twitter: @adforminsider > What is Adform: watch this short video > >=20 > Disclaimer: The information contained in this message and attachments = is intended solely for the attention and use of the named addressee and = may be confidential. If you are not the intended recipient, you are = reminded that the information remains the property of the sender. You = must not use, disclose, distribute, copy, print or rely on this e-mail. = If you have received this message in error, please contact the sender = immediately and irrevocably delete this message and any copies. > =20 > From: Brandon Williams [mailto:driftx@gmail.com]=20 > Sent: Thursday, May 24, 2012 02:35 > To: user@cassandra.apache.org > Subject: Re: Replication factor > =20 > On Wed, May 23, 2012 at 5:51 AM, Viktor Jevdokimov = wrote: > > When RF =3D=3D number of nodes, and you read at CL ONE you will = always be reading locally. > =93always be reading locally=94 =96 only if Dynamic Snitch is =93off=94.= With dynamic snitch =93on=94 request may be redirected to other node, = which may introduce latency spikes. > =20 > Actually it's preventing spikes, since if it won't read locally that = means the local replica is in worse shape than the rest (compacting, = repairing, etc.) > =20 > -Brandon=20 --Apple-Mail=_5D32A979-1EE9-4E79-83F1-0DD33B070EE5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Ah. The lack of page cache hits after compaction = makes sense. But I don't think the drastic effect it appears to have is = expected. Do you have an idea of how much slower local reads get = ?

If you are selecting coordinators based on token = ranges the DS is not as much. It still has some utility as the Digest = reads will be happening on other nodes and it should help with selecting = them. 

Thanks for the extra = info. 

Aaron

http://www.thelastpickle.com

On 25/05/2012, at 1:24 AM, Viktor Jevdokimov = wrote:

All = data is in the page cache. No repairs. Compactions not hitting disk for = read. CPU <50%. ParNew GC <100 ms in = average.
After = one compaction completes, new sstable is not in page cache, there may be = a disk usage spike before data is cached, so local reads gets slower for = a moment, comparing with other nodes. Redirecting almost all requests to = other nodes finally ends up with a huge latency spike almost on all = nodes, especially when ParNew GC may spike on one node (>200ms). We = call it =93cluster hiccup=94, when incoming and outgoing network traffic = drops for a moment.
 
And such hiccups happens several = times an hour, few seconds long. Playing with badness threshold did not = gave a lot better results, but turning DS off completely fixed all = problems with latencies, node spikes, cluster hiccups and network = traffic drops.
In = our case, our client is selecting endpoints for a key by calculating a = token, so we always hit a replica.
Best regards = / Pagarbiai
Viktor = Jevdokimov
Senior = Developer

What is = Adform: watch this short = video

 
Your = experience is when using CL ONE the Dynamic Snitch is moving local reads = off to other nodes and this is causing spikes in read latency = ? 
Did you notice what was = happening on the node for the DS to think it was so slow ? Was = compaction or repair going on ? 
 
Have you = played with the badness threshold Aaron = Morton