Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F1F2A188B7 for ; Wed, 2 Dec 2015 14:22:17 +0000 (UTC) Received: (qmail 53706 invoked by uid 500); 2 Dec 2015 14:22:15 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 53666 invoked by uid 500); 2 Dec 2015 14:22:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53656 invoked by uid 99); 2 Dec 2015 14:22:14 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2015 14:22:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 734D1180975 for ; Wed, 2 Dec 2015 14:22:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.899 X-Spam-Level: ** X-Spam-Status: No, score=2.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id bK8qGhFcXu15 for ; Wed, 2 Dec 2015 14:22:08 +0000 (UTC) Received: from mail-yk0-f171.google.com (mail-yk0-f171.google.com [209.85.160.171]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id CA75D42ABE for ; Wed, 2 Dec 2015 14:22:07 +0000 (UTC) Received: by ykba77 with SMTP id a77so47662410ykb.2 for ; Wed, 02 Dec 2015 06:22:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=8Ow/Q7iab/CzTct+2Hv5PBvlhKfJ7hqdX3ZCR7XgayY=; b=WqZsOfvhsbG1jJviKxlSx9g47L9CUQ4SHv2CZz94kT6u492S59EUw1FvQazk5d9sYG C2n0A3rLdx8jF2AXvb0G5QurjsScs+48nvByc8FmaknRHR20ya72BJH+bXonWMfyNGOR A83GmMX8RP3gyf0DMfad4pgGT8HDEg0cfl2GfPSG7EGBko3IyRJnxPy6Tt7KavrpFQx2 GGuVdiUTE6WrPB7+xJaTTWv/todg2YSVbxIT7eFZK7UbNF17S65ZaagzRFUhLN2rj3Tn xTti4j8BS8wgywL/aiA26a3ucKThgeMQnSwnLCExD2qxPYNYgeOm8Q5SD15znFR2TKrg gBnQ== X-Received: by 10.129.128.135 with SMTP id q129mr2140619ywf.325.1449066127321; Wed, 02 Dec 2015 06:22:07 -0800 (PST) MIME-Version: 1.0 Received: by 10.129.44.10 with HTTP; Wed, 2 Dec 2015 06:21:47 -0800 (PST) In-Reply-To: References: From: DuyHai Doan Date: Wed, 2 Dec 2015 18:21:47 +0400 Message-ID: Subject: Re: cassandra reads are unbalanced To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=94eb2c0329c4c69e1a0525eaffdb --94eb2c0329c4c69e1a0525eaffdb Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Which Consistency level do you use for reads ? ONE ? Are you reading from only DC1 or from both DC ? What is the LoadBalancingStrategy you have configured for your driver ? TokenAware wrapped on DCAwareRoundRobin ? On Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen wrote: > Hey all, > > > > Thanks for taking the time to help. > > > > So we have 6 cassandra nodes in 2 Data Centers. > > Both Data Centers have a replication of 3 =E2=80=93 so all nodes have all= the data. > > > > Over the last 2 days we=E2=80=99ve noticed that data reads / writes has s= hifted > from balanced to unbalanced > > (Nodetool status still shows 100% ownership on every node, with similar > sizes) > > > > > > For Example > > > > We monitor the number of reads / writes of every table via the cassandra > JMX metrics. (cassandra.db.read_count) > > Over the last hour of this run > > > > Reads > > Node 1 (DC1) =3D 1.79k (seeder) > > Node 2 (DC1) =3D 1.92k > > Node 3 (DC1) =3D 1.97k > > > > Node 1 (DC2) =3D 2.90k (seeder) > > Node 2 (DC2) =3D 1.76k > > Node 3 (DC2) =3D 1.19k > > > > As you see on DC1, everything is pretty well balanced, but on DC2 the > reads favour Node1 over Node 3. > > I ran a nodetool repair yesterday =E2=80=93 ran for 6 hours and when comp= leted > didn=E2=80=99t change the read balance. > > > > Write levels are similar on DC2, but not as bad a reads. > > > > Anyone any suggestion on how to rebalance? I=E2=80=99m thinking maybe run= ning a > nodetool cleanup in case some of the keys have shifted? > > > > Regards > > Stephen Walsh > > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have receiv= ed > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > --94eb2c0329c4c69e1a0525eaffdb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Which Consistency level do you use for reads ? ONE ? Are y= ou reading from only DC1 or from both DC ?

What is the LoadBala= ncingStrategy you have configured for your driver ? TokenAware wrapped on D= CAwareRoundRobin ?



<= br>

On= Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen <Stephen.Walsh@aspect= .com> wrote:

Hey all,

=C2=A0

Thanks for taking the time to help.

=C2=A0

So we have 6 cassandra nodes in 2 Data Centers.

Both Data Centers have a replication of 3 =E2=80=93 = so all nodes have all the data.

=C2=A0

Over the last 2 days we=E2=80=99ve noticed that data= reads / writes has shifted from balanced to unbalanced

(Nodetool status still shows 100% ownership on every= node, with similar sizes)

=C2=A0

=C2=A0

For Example

=C2=A0

We monitor the number of reads / writes of every tab= le via the cassandra JMX metrics. (cassandra.db.read_count)

Over the last hour of this run

=C2=A0

Reads

Node 1 (DC1) =C2=A0=3D =C2=A01.79k=C2=A0=C2=A0=C2=A0= (seeder)

Node 2 (DC1) =C2=A0=3D =C2=A01.92k=

Node 3 (DC1) =C2=A0=3D =C2=A01.97k=

=C2=A0

Node 1 (DC2) =C2=A0=3D =C2=A02.90k= =C2=A0=C2=A0 (seeder)

Node 2 (DC2) =C2=A0=3D =C2=A01.76k=

Node 3 (DC2) =C2=A0=3D =C2=A01.19k=

=C2=A0

As you see on DC1, everything is pretty well balance= d, but on DC2 the reads favour Node1 over Node 3.

I ran a nodetool repair yesterday =E2=80=93 ran for = 6 hours and when completed didn=E2=80=99t change the read balance.

=C2=A0

Write levels are similar on=C2=A0 DC2, but not as ba= d a reads.

=C2=A0

Anyone any suggestion on how to rebalance? I=E2=80= =99m thinking maybe running a nodetool cleanup in case some of the keys hav= e shifted?

=C2=A0

Regards

Stephen Walsh

=C2=A0

=C2=A0

This email (including any attachments) is proprietary to Aspect Software, I= nc. and may contain information that is confidential. If you have received = this message in error, please do not read, copy or forward this message. Pl= ease notify the sender immediately, delete it from your system and destroy any copies. You may not further dis= close or distribute this email or its attachments.

--94eb2c0329c4c69e1a0525eaffdb--