Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A2207188E3 for ; Fri, 4 Dec 2015 16:46:06 +0000 (UTC) Received: (qmail 13588 invoked by uid 500); 4 Dec 2015 16:45:57 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13553 invoked by uid 500); 4 Dec 2015 16:45:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13543 invoked by uid 99); 4 Dec 2015 16:45:57 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Dec 2015 16:45:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A680DCCDA4 for ; Fri, 4 Dec 2015 16:45:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.1 X-Spam-Level: ***** X-Spam-Status: No, score=5.1 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=3, KAM_LINEPADDING=1.2, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id GzOg4rM_WiP6 for ; Fri, 4 Dec 2015 16:45:42 +0000 (UTC) Received: from mail-vk0-f54.google.com (mail-vk0-f54.google.com [209.85.213.54]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 43D952059E for ; Fri, 4 Dec 2015 16:45:42 +0000 (UTC) Received: by vkay187 with SMTP id y187so68795718vka.3 for ; Fri, 04 Dec 2015 08:45:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=zsmsx7pRIrS6PYVwCXyHhADPDScPDmGs5CZggdtpYFw=; b=oedkp8CfJKtB0FFiNBeW32BvIkc7N/jd7sReheBmHzmJXOYZQvg9uHDdBUWPaVsYRT MoPggOEgx/3IE9u/SMTXU6sYYVyEiKmRHZT0OBU3gyRbwaCxt4D7gSc12F3OnjXk1MhE Q0vXYWxATKhhgQPnL/V+AXIMdFuKfcvyGOLJ59I75zoaQQGDSrKcXrKWlTMMoRCCunrN 6PBj50dDE8BaRx4AvKzzwMCmLO258Ir8Y5xNbVS5XkQQsOkL+FIkCCxRpdEW4gQr4VAm KxU8RbBfy1gxjmPFyA6aWGPALGKaDFRPgCfoJsGlKoW2EH940SukGOCJPYWFfd+Mhw+O 5L4A== MIME-Version: 1.0 X-Received: by 10.31.5.132 with SMTP id 126mr10905017vkf.107.1449247541151; Fri, 04 Dec 2015 08:45:41 -0800 (PST) Received: by 10.31.47.137 with HTTP; Fri, 4 Dec 2015 08:45:41 -0800 (PST) In-Reply-To: References: Date: Fri, 4 Dec 2015 11:45:41 -0500 Message-ID: Subject: Re: cassandra reads are unbalanced From: Jack Krupansky To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11440f82e1cdf60526153ce7 --001a11440f82e1cdf60526153ce7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks for the elaboration. A few more questions... Is there only a single thread in each client or are there multiple threads doing reading in parallel? IOW, does a read need to complete before the next read is issued. What client Cassandra driver are you using? Java? What does your connection code look like, say compared to the example in the doc: http://docs.datastax.com/en/developer/java-driver/2.0/java-driver/quick_sta= rt/qsSimpleClientCreate_t.html Just to make sure it really is connecting only to the local cluster and using round robin and whether it is token aware. -- Jack Krupansky On Fri, Dec 4, 2015 at 10:51 AM, Walsh, Stephen wrote: > Thanks for your input, but I think I=E2=80=99ve already answered most of = your > questions. > > > > > > How many clients do you have performing reads? > > > > ------------------ > > On Wed, Dec 2, 2015 at 6:44 PM, Walsh, Stephen > wrote > > =E2=80=A6. > > There are 2 application (1 for each DC) who read and write at the same > rate to their local DC > > =E2=80=A6. > > -------------------- > > > > > > > > > > > > > > Is your load balancer in front of your clients or between your clients an= d > Cassandra? > > > > ------------------ > > On Thu, Dec 3, 2015 at 4:58 AM, Walsh, Stephen > wrote: > > =E2=80=A6 > > our production applications are behind a round robin load balancer > > =E2=80=A6 > > ------------------ > > > > No Load Balancers talk to cassandra =E2=80=93 I=E2=80=99m only mentioning= this to show > that the writes / read are evenly distributed over the 2 DC=E2=80=99s > > > > > > > > > > > > > > Does Node1 of DC2 have the exact same configuration of hardware of the > other nodes > > Yes > > > > > > > > > > > > Is it in the same rack > > It=E2=80=99s in AWS =E2=80=93 but we have it configured via the GossipPro= perytFileSnitch > that they are all on unique racks > > > > > > > > > > > > > > Maybe your load balancer thinks that node is more capable and handles > requests faster so that it looks less loaded than the other two nodes > > Unlikely, it=E2=80=99s all TCP SSL pass though connections. It doesn=E2= =80=99t balance on > load, it just round robins each request > > > > > > > > > > > > You might also check the read counts after a very short interval of time > to see if Node1 is uniformly getting more requests or just occasionally > > ------------------ > > On Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen > wrote =E2=80=A6 > > We monitor the number of reads / writes of every table via the cassandra > JMX metrics. (cassandra.db.read_count) > > =E2=80=A6 > > ------------------ > > We can only monitor in 1 hour moving window > > > > > > > > > > Maybe the other two nodes are in a different rack that occasionally has > net connectivity issues > > Unlikely seems its AWS > > > > > > > > > > > > > > *From:* Jack Krupansky [mailto:jack.krupansky@gmail.com] > *Sent:* 03 December 2015 16:11 > > *To:* user@cassandra.apache.org > *Subject:* Re: cassandra reads are unbalanced > > > > How many clients do you have performing reads? > > > > Is your load balancer in front of your clients or between your clients an= d > Cassandra? > > > > Does Node1 of DC2 have the exact same configuration of hardware of the > other nodes? Is it in the same rack? Maybe your load balancer thinks that > node is more capable and handles requests faster so that it looks less > loaded than the other two nodes. > > > > You might also check the read counts after a very short interval of time > to see if Node1 is uniformly getting more requests or just occasionally. > Maybe the other two nodes are in a different rack that occasionally has n= et > connectivity issues so that the requests get diverted by the client/load > balancer to Node1 during those times. > > > > > -- Jack Krupansky > > > > On Thu, Dec 3, 2015 at 4:58 AM, Walsh, Stephen > wrote: > > Thanks but keep in mind that both DC should be getting the same load, our > production applications are behind a round robin load balancer =E2=80=93 = so each > one our local application talk to its local Cassandra DataCenter. > > > > It took about 4 hours but the nodetool cleanup eventually balanced all > nodes > > > > *From:* DuyHai Doan [mailto:doanduyhai@gmail.com] > *Sent:* 02 December 2015 16:27 > > > *To:* user@cassandra.apache.org > *Subject:* Re: cassandra reads are unbalanced > > > > If you're using the Java driver with LOCAL_ONE and the default load > balancing strategy (TokenAware wrapped on DCAwareRoundRobin), the > driver will always select the primary replica. To change this behavior an= d > introduce some randomness so that non primary replicas get a chance to > serve a read: > > > > new TokenAwarePolicy(new DCAwareRoundRobinPolicy("local_DC"), true). > > > > The second parameter (true) asks the TokenAware policy to "shuffle" > replica on each request to avoid always returning the primary replica. > > > > On Wed, Dec 2, 2015 at 6:44 PM, Walsh, Stephen > wrote: > > Very good questions. > > > > We have reads and writes at LOCAL_ONE. > > There are 2 application (1 for each DC) who read and write at the same > rate to their local DC > > (All reads / writes started all perfectly even and degraded over time) > > > > We use DCAwareRoundRobin policy > > > > On update on the nodetool cleanup =E2=80=93 it has help but hasn=E2=80=99= t balanced all > nodes. Node 1 on DC2 is still quite high > > > > Node 1 (DC1) =3D 1.35k (seeder) > > Node 2 (DC1) =3D 1.54k > > Node 3 (DC1) =3D 1.45k > > > > Node 1 (DC2) =3D 2.06k (seeder) > > Node 2 (DC2) =3D 1.38k > > Node 3 (DC2) =3D 1.43k > > > > > > *From:* DuyHai Doan [mailto:doanduyhai@gmail.com] > *Sent:* 02 December 2015 14:22 > *To:* user@cassandra.apache.org > *Subject:* Re: cassandra reads are unbalanced > > > > Which Consistency level do you use for reads ? ONE ? Are you reading from > only DC1 or from both DC ? > > What is the LoadBalancingStrategy you have configured for your driver ? > TokenAware wrapped on DCAwareRoundRobin ? > > > > > > > > > > > > On Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen > wrote: > > Hey all, > > > > Thanks for taking the time to help. > > > > So we have 6 cassandra nodes in 2 Data Centers. > > Both Data Centers have a replication of 3 =E2=80=93 so all nodes have all= the data. > > > > Over the last 2 days we=E2=80=99ve noticed that data reads / writes has s= hifted > from balanced to unbalanced > > (Nodetool status still shows 100% ownership on every node, with similar > sizes) > > > > > > For Example > > > > We monitor the number of reads / writes of every table via the cassandra > JMX metrics. (cassandra.db.read_count) > > Over the last hour of this run > > > > Reads > > Node 1 (DC1) =3D 1.79k (seeder) > > Node 2 (DC1) =3D 1.92k > > Node 3 (DC1) =3D 1.97k > > > > Node 1 (DC2) =3D 2.90k (seeder) > > Node 2 (DC2) =3D 1.76k > > Node 3 (DC2) =3D 1.19k > > > > As you see on DC1, everything is pretty well balanced, but on DC2 the > reads favour Node1 over Node 3. > > I ran a nodetool repair yesterday =E2=80=93 ran for 6 hours and when comp= leted > didn=E2=80=99t change the read balance. > > > > Write levels are similar on DC2, but not as bad a reads. > > > > Anyone any suggestion on how to rebalance? I=E2=80=99m thinking maybe run= ning a > nodetool cleanup in case some of the keys have shifted? > > > > Regards > > Stephen Walsh > > > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have receiv= ed > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have receiv= ed > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have receiv= ed > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have receiv= ed > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > --001a11440f82e1cdf60526153ce7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks for the elaboration. A few more questions...
Is there only a single thread in each client or are there mult= iple threads doing reading in parallel? IOW, does a read need to complete b= efore the next read is issued.

What client Cassand= ra driver are you using? Java?

What does your conn= ection code look like, say compared to the example in the doc:
http://docs.datastax.com/en/develo= per/java-driver/2.0/java-driver/quick_start/qsSimpleClientCreate_t.html=

Just to make sure it really is connecting only to= the local cluster and using round robin and whether it is token aware.
=


-- Jack Krupansky

On Fri, Dec 4, 2015 at 10:51 AM, Walsh, Step= hen <Stephen.Walsh@aspect.com> wrote:

Thanks for your input, but I think I=E2=80=99ve alre= ady answered most of your questions.

=C2=A0

=C2=A0

How many clients do you have performing reads?

=C2=A0

------------------

On Wed, Dec 2, 2015 at 6:44 PM, Walsh= , Stephen <Stephen.Walsh@aspect.co= m> wrote

=E2=80=A6.

There are 2 application (1 for each D= C) who read and write at the same rate to their local DC

=E2=80=A6.

--------------------

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

Is your load balancer in front of your clients or be= tween your clients and Cassandra?

=C2=A0

------------------

On Thu, Dec 3, 2015 at 4:58 AM, Walsh= , Stephen <Stephen.Walsh@aspect.co= m> wrote:

=E2=80=A6=

our production applications are behin= d a round robin load balancer

=E2=80=A6=

------------------

=C2=A0

No Load Balancers talk to cassandra = =E2=80=93 I=E2=80=99m only mentioning this to show that the writes / read a= re evenly distributed over the 2 DC=E2=80=99s

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

Does Node1 of DC2 have the exact same configuration = of hardware of the other nodes

Yes

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

Is it in the same rack

It=E2=80=99s in AWS =E2=80=93 = but we have it configured via the GossipProperytFileSnitch that they are al= l on unique racks

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

Maybe your load balancer thinks that node is more ca= pable and handles requests faster so that it looks less loaded than the oth= er two nodes

Unlikely, it=E2=80=99s all TCP= SSL pass though connections. It doesn=E2=80=99t balance on load, it just r= ound robins each request

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

You might also check the read counts after a very sh= ort interval of time to see if Node1 is uniformly getting more requests or = just occasionally

------------------

On Wed, Dec 2, 2015 at 3:36 PM, Walsh= , Stephen <Stephen.Walsh@aspect.co= m> wrote =E2=80=A6

We monitor the number of reads / writ= es of every table via the cassandra JMX metrics. (cassandra.db.read_count)<= u>

=E2=80=A6=

------------------

We can only monitor in 1 hour moving = window

=C2=A0

=C2=A0

=C2=A0

=C2=A0

Maybe the other two nodes are in a different rack th= at occasionally has net connectivity issues

Unlikely seems its AWS<= u>

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

From: = Jack Krupansky [mailto:jack.krupansky@gmail.com]
Sent: 03 December 2015 16:11


To: u= ser@cassandra.apache.org
Subject: Re: cassandra reads are unbalanced

=C2=A0

How many clients do you have performing reads?

=C2=A0

Is your load balancer in front of your clients or be= tween your clients and Cassandra?

=C2=A0

Does Node1 of DC2 have the exact same configuration = of hardware of the other nodes? Is it in the same rack? Maybe your load bal= ancer thinks that node is more capable and handles requests faster so that = it looks less loaded than the other two nodes.

=C2=A0

You might also check the read counts after a very sh= ort interval of time to see if Node1 is uniformly getting more requests or = just occasionally. Maybe the other two nodes are in a different rack that o= ccasionally has net connectivity issues so that the requests get diverted by the client/load balancer to Node1 dur= ing those times.

=C2=A0


-- Jack Krupansky

=C2=A0

On Thu, Dec 3, 2015 at 4:58 AM, Walsh, Stephen <<= a href=3D"mailto:Stephen.Walsh@aspect.com" target=3D"_blank">Stephen.Walsh@= aspect.com> wrote:

Thanks but keep in mind that both DC = should be getting the same load, our production applications are behind a round robin load balancer =E2=80=93 so each one our local applica= tion talk to its local Cassandra DataCenter.

=C2=A0

It took about 4 hours but the nodetoo= l cleanup eventually balanced all nodes

=C2=A0

From: = DuyHai Doan [mailto:doa= nduyhai@gmail.com]
Sent: 02 December 2015 16:27


To: u= ser@cassandra.apache.org
Subject: Re: cassandra reads are unbalanced

=C2=A0

If you're using the Java driver with LOCAL_ONE a= nd the default load balancing strategy (TokenAware wrapped on=C2=A0DCAwareR= oundRobin), the driver=C2=A0will always select the primary replica. To change this behavior and introduce some randomness so that non primary = replicas get a chance to serve a read:

=C2=A0

new TokenAwarePolicy(new DCAwareRoundRobinPolicy(&qu= ot;local_DC"), true).

=C2=A0

The second parameter (true) asks the TokenAware poli= cy to "shuffle" replica on each request to avoid always returning= the primary replica.

=C2=A0

On Wed, Dec 2, 2015 at 6:44 PM, Walsh, Stephen <<= a href=3D"mailto:Stephen.Walsh@aspect.com" target=3D"_blank">Stephen.Walsh@= aspect.com> wrote:

Very good questions.=

=C2=A0

We have reads and writes at LOCAL_ONE= .

There are 2 application (1 for each D= C) who read and write at the same rate to their local DC

(All reads / writes started all perfe= ctly even and degraded over time)

=C2=A0

We use DCAwareRoundRobin policy

=C2=A0

On update on the nodetool cleanup =E2= =80=93 it has help but hasn=E2=80=99t balanced all nodes. Node 1 on DC2 is = still quite high

=C2=A0

Node 1 (DC1) =C2=A0=3D =C2=A01.35k= =C2=A0=C2=A0=C2=A0 (seeder)

Node 2 (DC1) =C2=A0=3D =C2=A01.54k=

Node 3 (DC1) =C2=A0=3D =C2=A01.45k=

=C2=A0

Node 1 (DC2) =C2=A0=3D =C2=A02.06k= =C2=A0=C2=A0 (seeder)

Node 2 (DC2) =C2=A0=3D =C2=A01.38k=

Node 3 (DC2) =C2=A0=3D =C2=A01.43k=

=C2=A0

=C2=A0

From: = DuyHai Doan [mailto:doa= nduyhai@gmail.com]
Sent: 02 December 2015 14:22
To: u= ser@cassandra.apache.org
Subject: Re: cassandra reads are unbalanced

=C2=A0

Which Consistency lev= el do you use for reads ? ONE ? Are you reading from only DC1 or from both = DC ?

What is the LoadBalancingStrategy you have configure= d for your driver ? TokenAware wrapped on DCAwareRoundRobin ?=

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

On Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen <<= a href=3D"mailto:Stephen.Walsh@aspect.com" target=3D"_blank">Stephen.Walsh@= aspect.com> wrote:

Hey all,

=C2=A0

Thanks for taking the time to help.

=C2=A0

So we have 6 cassandra nodes in 2 Data Centers.

Both Data Centers have a replication of 3 =E2=80=93 = so all nodes have all the data.

=C2=A0

Over the last 2 days we=E2=80=99ve noticed that data= reads / writes has shifted from balanced to unbalanced

(Nodetool status still shows 100% ownership on every= node, with similar sizes)

=C2=A0

=C2=A0

For Example

=C2=A0

We monitor the number of reads / writes of every tab= le via the cassandra JMX metrics. (cassandra.db.read_count)

Over the last hour of this run

=C2=A0

Reads

Node 1 (DC1) =C2=A0=3D =C2=A01.79k=C2=A0=C2=A0=C2=A0= (seeder)

Node 2 (DC1) =C2=A0=3D =C2=A01.92k=

Node 3 (DC1) =C2=A0=3D =C2=A01.97k=

=C2=A0

Node 1 (DC2) =C2=A0=3D =C2=A02.90k= =C2=A0=C2=A0 (seeder)

Node 2 (DC2) =C2=A0=3D =C2=A01.76k=

Node 3 (DC2) =C2=A0=3D =C2=A01.19k=

=C2=A0

As you see on DC1, everything is pretty well balance= d, but on DC2 the reads favour Node1 over Node 3.

I ran a nodetool repair yesterday =E2=80=93 ran for = 6 hours and when completed didn=E2=80=99t change the read balance.

=C2=A0

Write levels are similar on=C2=A0 DC2, but not as ba= d a reads.

=C2=A0

Anyone any suggestion on how to rebalance? I=E2=80= =99m thinking maybe running a nodetool cleanup in case some of the keys hav= e shifted?

=C2=A0

Regards

Stephen Walsh

=C2=A0

=C2=A0

This email (including any attachments) is proprietar= y to Aspect Software, Inc. and may contain information that is confidential= . If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immedi= ately, delete it from your system and destroy any copies. You may not furth= er disclose or distribute this email or its attachments.

=C2=A0

This email (including any attachments) is proprietar= y to Aspect Software, Inc. and may contain information that is confidential= . If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immedi= ately, delete it from your system and destroy any copies. You may not furth= er disclose or distribute this email or its attachments.

=C2=A0

This email (including any attachments) is proprietar= y to Aspect Software, Inc. and may contain information that is confidential= . If you have received this message in error, please do not read, copy or f= orward this message. Please notify the sender immediately, delete it from your system and destroy any copies.= You may not further disclose or distribute this email or its attachments.

=C2=A0

This email (including any attachments) is proprietary to Aspect Software, I= nc. and may contain information that is confidential. If you have received = this message in error, please do not read, copy or forward this message. Pl= ease notify the sender immediately, delete it from your system and destroy any copies. You may not further dis= close or distribute this email or its attachments.

--001a11440f82e1cdf60526153ce7--