Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 428B6187C2 for ; Tue, 9 Jun 2015 17:29:44 +0000 (UTC) Received: (qmail 2022 invoked by uid 500); 9 Jun 2015 17:29:41 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 1976 invoked by uid 500); 9 Jun 2015 17:29:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 1962 invoked by uid 99); 9 Jun 2015 17:29:41 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2015 17:29:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D2094CCC91 for ; Tue, 9 Jun 2015 17:29:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.901 X-Spam-Level: **** X-Spam-Status: No, score=4.901 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLYTO_END_DIGIT=0.25, HTML_MESSAGE=3, KAM_WEIRDTRICK1=1.5, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.co.in Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id vFHBgUoA3lRi for ; Tue, 9 Jun 2015 17:29:25 +0000 (UTC) Received: from nm17-vm3.bullet.mail.sg3.yahoo.com (nm17-vm3.bullet.mail.sg3.yahoo.com [106.10.149.82]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id B773847BD9 for ; Tue, 9 Jun 2015 17:29:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.co.in; s=s2048; t=1433870894; bh=Iz47drVZMwFIiz5glw022F6N50hOWH6OPb6GWEJsHh4=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=mUAyxvBY+asGszGR1MBuf+OBdsIhBwD5QjiND2AMkk3lVjmu+U692a3bYZqXZu59uosojeTgLJYpE113c+i0FRQ25gi3eobdS6dr7ikiFIr8VdLgCS9wu7yMe3kPpt1qRxDiD+mKIKLrTt8ZmftSh8ASXhULJ5SiCPOhpDZCaGBoegFLUydJJlL3raca9ks85tQPApg2miU5aeOLgD0ar89Tfn2EuLuviNa30HAejVdEPdeD0FdTm5AdKBKQacQyjgRIkrTM6gVWq9C5+4dd1qXBdandovTvsY+LIdU9cDCY+FZfJ42Ngnr7wEDNeAbmZX7HiXM9caX6bYV5Ak6BaQ== Received: from [106.10.166.125] by nm17.bullet.mail.sg3.yahoo.com with NNFMP; 09 Jun 2015 17:28:14 -0000 Received: from [106.10.151.253] by tm14.bullet.mail.sg3.yahoo.com with NNFMP; 09 Jun 2015 17:28:14 -0000 Received: from [127.0.0.1] by omp1002.mail.sg3.yahoo.com with NNFMP; 09 Jun 2015 17:28:14 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 307199.66800.bm@omp1002.mail.sg3.yahoo.com X-YMail-OSG: zUWACZgVM1mmb5Jq9e4NKYbjmzEIbXghzCcy5on0SLl7Obnlb52UeQeN6IuGIix gDvdcCkzSCP3KxD85PCZKyapo2gOPEQD5UgW98i1jAHt82aIazZHmvOBUkw3Cwqd.PMnuxnZIcnF 1spOOOy0md5rLzs1rA6sSAzpR.kIDgLQs49XfnTUnOuATMkfyxukXujHwKIvFMTaJQxLL.PNfIA5 Sz2THnn76ahBBVVVlVzQQB8OEVPlLllHInH.FGyyZTCs0KP47K9tK7Dq37U.G2dq.X0NMLnsi1MN XmYSNL8DHjbmphb77CvxuG0slW.4JG9DWhaq32ys_fTBvJXYQy6SBrFqhOFTQQhuSZzS0ocY49Ug CCqGaVWyGoI7cH0fKLBM1dJoj4gqSeIZ6xMeZ_fqup.CWcdTIEzaWJg_JtolOpW58CfQoVcntdJf LOxB0wSBwbt4Ks7cYTJv7PD5qcBCoETptIdv3zo4m8qW82VvyA7JVjmXGFO9PN4LGrxjwhyZ3bBw WAtTOnWfgqZBv Received: by 106.10.196.183; Tue, 09 Jun 2015 17:28:13 +0000 Date: Tue, 9 Jun 2015 17:28:13 +0000 (UTC) From: Anuj Wadehra Reply-To: Anuj Wadehra To: "user@cassandra.apache.org" Message-ID: <1340805617.4928569.1433870893432.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: References: Subject: Re: Hundreds of sstables after every Repair MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_4928568_1075139613.1433870893419" ------=_Part_4928568_1075139613.1433870893419 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Yes. We use NTP. We also thought that drift is creating problems. Our NTP O= utput is as under: [root@node1 ~]# ntpq -p =C2=A0=C2=A0=C2=A0=C2=A0 remote=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 refid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 st t when poll reach= =C2=A0=C2=A0 delay=C2=A0=C2=A0 offset=C2=A0 jitter =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 2 u=C2= =A0 237 1024=C2=A0 377=C2=A0=C2=A0=C2=A0 1.199=C2=A0=C2=A0=C2=A0 0.062=C2= =A0=C2=A0 0.554 *10.x.x.x=C2=A0=C2=A0=C2=A0 10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 2 u=C2=A0 178 = 1024=C2=A0 377=C2=A0=C2=A0=C2=A0 0.479=C2=A0=C2=A0 -0.350=C2=A0=C2=A0 0.626 =C2=A0 =C2=A0 =C2=A0 [root@node2 ~]# ntpq -p =C2=A0=C2=A0=C2=A0=C2=A0 remote=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 refid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 st t when poll reach= =C2=A0=C2=A0 delay=C2=A0=C2=A0 offset=C2=A0 jitter =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 2 u=C2= =A0 124 1024=C2=A0 377=C2=A0=C2=A0=C2=A0 0.939=C2=A0=C2=A0 -0.001=C2=A0=C2= =A0 0.614 *10.x.x.x=C2=A0=C2=A0=C2=A0 10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 2 u=C2=A0 722 = 1024=C2=A0 377=C2=A0=C2=A0=C2=A0 0.567=C2=A0=C2=A0 -0.241=C2=A0=C2=A0 0.585 =C2=A0 =C2=A0 =C2=A0 [root@node3 ~]# ntpq -p =C2=A0=C2=A0=C2=A0=C2=A0 remote=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 refid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 st t when poll reach= =C2=A0=C2=A0 delay=C2=A0=C2=A0 offset=C2=A0 jitter =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 2 u=C2= =A0 514 1024=C2=A0 377=C2=A0=C2=A0=C2=A0 0.716=C2=A0=C2=A0 -0.103=C2=A0=C2= =A0 1.315 *10.x.x.x=C2=A0=C2=A0=C2=A0 10.x.x.x=C2=A0=C2=A0=C2=A0=C2=A0 2 u=C2=A0=C2= =A0 21 1024=C2=A0 377=C2=A0=C2=A0=C2=A0 0.402=C2=A0=C2=A0 -0.262=C2=A0=C2= =A0 1.070 =C2=A0 =C2=A0 ***IPs are masked ThanksAnuj Wadehra =20 On Tuesday, 9 June 2015 9:12 PM, Carlos Rolo wrote: =20 Hello, Do you have your clocks synced across your cluster? Are you using NTP and h= ave it properly configured? Sometimes clock out of sync can trigger weird behaviour. Regards, Carlos Juzarte RoloCassandra Consultant=C2=A0Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterol= oMobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649www.pythian.com On Tue, Jun 9, 2015 at 5:11 PM, Anuj Wadehra wrote= : | We were facing dropped mutations earlier and we increased flush writers. = Now there are no dropped mutations in tpstats. To repair the damaged vnodes= / inconsistent data we executed repair -pr on all nodes. Still, we see the= same problem.=C2=A0 When we analyze repair logs we see 2 strange things: 1. "Out of sync" ranges for cf which are not being actively being written/u= pdated while the repair is going on. When we repaired all data by repair -p= r on all nodes, why out of sync data? 2. For some cf , repair logs shows that all ranges are consistent. Still we= get so many sstables created during repair. When everything is in sync , w= hy repair creates tiny sstables to repair data? ThanksAnuj Wadehra Sent from Yahoo Mail on Android=20 | From:"Ken Hancock" Date:Tue, 9 Jun, 2015 at 8:24 pm Subject:Re: Hundreds of sstables after every Repair I think this came up recently in another thread.=C2=A0 If you're getting l= arge numbers of SSTables after repairs, that means that your nodes are dive= rging from the keys that they're supposed to be having.=C2=A0 Likely you're= dropping mutations.=C2=A0 Do a nodetool tpstats on each of your nodes and = look at the mutation droppped counters.=C2=A0 If you're seeing dropped mess= age, my money you have a non-zero FlushWriter "All time blocked" stat which= is causing mutations to be dropped. On Tue, Jun 9, 2015 at 10:35 AM, Anuj Wadehra wrot= e: | Any suggestions or comments on this one? ThanksAnuj Wadehra Sent from Yahoo Mail on Android =20 | From:"Anuj Wadehra" Date:Sun, 7 Jun, 2015 at 1:54 am Subject:Hundreds of sstables after every Repair Hi, We are using 2.0.3 and vnodes. After every repair -pr operation=C2=A0 50+ t= iny sstables( <10K) get created. And these sstables never get compacted due= to coldness issue. I have raised https://issues.apache.org/jira/browse/CAS= SANDRA-9146 for this issue but I have been told to upgrade. Till we upgrade= to latest 2.0.x , we are stuck. Upgrade takes time, testing and planning i= n Production systems :( I have observed that even if vnodes are NOT damaged, hundreds of tiny sstab= les are created during repair for a wide row CF. This is beyond my understa= nding. If everything is consistent, and for the entire repair process Cassa= ndra is saying "Endpoints /x.x.x.x and /x.x.x.y are consistent for ". W= hats the need of creating sstables? Is there any alternative to regular major compaction to deal with situation= ?=20 ThanksAnuj Wadehra | | |=20 |=20 | |=20 | |=20 | | | -- ------=_Part_4928568_1075139613.1433870893419 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes. We use NTP. We also thought that drift is creating pro= blems. Our NTP Output is as under:

= [root@node1 ~]# ntpq -p
     remote   =         refid    &nb= sp; st t when poll reach   delay   offset  jitter
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
+10.x.x.x     10.x.x.x &= nbsp;   2 u  237 1024  377    1.199 = ;   0.062   0.554
*10.x.x.x    10.x.x.x  &= nbsp;  2 u  178 1024  377    0.479  = ; -0.350   0.626
 
 
 
[root@node2 ~]# ntpq -p
     remote   =         refid    &nb= sp; st t when poll reach   delay   offset  jitter
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
+10.x.x.x     10.x.x.x &= nbsp;   2 u  124 1024  377    0.939 = ;  -0.001   0.614
*10.x.x.x    10.x.x.x  &= nbsp;  2 u  722 1024  377    0.567  = ; -0.241   0.585
 
 
 
[root@node3 ~]# ntpq -p
     remote   =         refid    &nb= sp; st t when poll reach   delay   offset  jitter
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
+10.x.x.x     10.x.x.x &= nbsp;   2 u  514 1024  377    0.716 = ;  -0.103   1.315
*10.x.x.x    10.x.x.x  &= nbsp;  2 u   21 1024  377    0.402 =   -0.262   1.070
 
 
***IPs are masked

Thanks
Anuj Wadehra



On= Tuesday, 9 June 2015 9:12 PM, Carlos Rolo <rolo@pythian.com> wrote:<= br>


Hello,

Do you have your clocks synced across your cluster? Are y= ou using NTP and have it properly configured?

Sometimes clock out of sync can trigger weird behaviour.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant
 
Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
=

On Tue, Jun 9, 2015 = at 5:11 PM, Anuj Wadehra <anujw_2003@yahoo.co.in> wrote:=
We were facing dropped mutations earlier and we= increased flush writers. Now there are no dropped mutations in tpstats. To= repair the damaged vnodes / inconsistent data we executed repair -pr on al= l nodes. Still, we see the same problem. 

When we analyze repair logs we see 2 strange things:

1. "Out of sync" ranges for cf which are not being actively bei= ng written/updated while the repair is going on. When we repaired all data = by repair -pr on all nodes, why out of sync data?

2. For some cf , repair logs shows that all ranges are consistent. Still we get so many sstables created during repair= . When everything is in sync , why repair creates tiny sstables to repair d= ata?

Thanks
Anuj Wadehra

=
From:"Ken Hancock" <= ken.hancock@sc= hange.com>
Date:Tue, 9 Jun, 2015 at 8:24 pm=
Subject:Re: Hundreds of sstables after every Repa= ir

I think this came up recently in another thread.&n= bsp; If you're getting large numbers of SSTables after repairs, that means = that your nodes are diverging from the keys that they're supposed to be having.  Likely y= ou're dropping mutations.  Do a nodetool tpstats on each of your nodes= and look at the mutation droppped counters.  If you're seeing dropped= message, my money you have a non-zero FlushWriter "All time blocked" stat = which is causing mutations to be dropped.



On Tue, Jun 9, 2015 at = 10:35 AM, Anuj Wadehra <anujw_2003@yahoo.co.in> wrote:
Any suggestions or comments on this one?

Thanks
Anuj Wadehra

<= /tr>
From:"Anuj Wadehra" <anujw_2003@yahoo.co.in>
Date:Sun, 7 J= un, 2015 at 1:54 am
Subject:Hundreds of sstables a= fter every Repair

Hi,

We are using 2.0.3 and vnode= s. After every repair -pr operation  50+ tiny sstables( <10K) get c= reated. And these sstables never get compacted due to coldness issue. I hav= e raised https://issues.apache.org/= jira/browse/CASSANDRA-9146 for this issue but I have been told to upgra= de. Till we upgrade to latest 2.0.x , we are stuck. Upgrade takes time, tes= ting and planning in Production systems :(

I have observed that eve= n if vnodes are NOT damaged, hundreds of tiny sstables are created during r= epair for a wide row CF. This is beyond my understanding. If everything is consistent, and for the entire repair proc= ess Cassandra is saying "Endpoints /x.x.x.x and /x.x.x.y are consistent for= <CF>". Whats the need of creating sstables?
Is there any alternative to regular= major compaction to deal with situation?


Thanks
Anuj Wadehra







<= /td>


--




------=_Part_4928568_1075139613.1433870893419--