Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 33E4A200BEF for ; Wed, 4 Jan 2017 16:56:43 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 32320160B3A; Wed, 4 Jan 2017 15:56:43 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8FA1E160B39 for ; Wed, 4 Jan 2017 16:56:41 +0100 (CET) Received: (qmail 25815 invoked by uid 500); 4 Jan 2017 15:56:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 25801 invoked by uid 99); 4 Jan 2017 15:56:39 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2017 15:56:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 7A4A61A0874 for ; Wed, 4 Jan 2017 15:56:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.769 X-Spam-Level: ** X-Spam-Status: No, score=2.769 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_FONT_FACE_BAD=0.289, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=thelastpickle-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id fgEiBILfYvgI for ; Wed, 4 Jan 2017 15:56:36 +0000 (UTC) Received: from mail-vk0-f46.google.com (mail-vk0-f46.google.com [209.85.213.46]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 402555FD36 for ; Wed, 4 Jan 2017 15:56:36 +0000 (UTC) Received: by mail-vk0-f46.google.com with SMTP id y197so140244790vky.2 for ; Wed, 04 Jan 2017 07:56:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thelastpickle-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=MgnsQNNGmrb1/WPFPmQ/KLD2qYdy4H6vKjniKjTVkwc=; b=s0IAEh+llBCcfQ7IOYFTucDqm4bzq/ITyaaMWUF96cBSyseaWzt3IzK8xiXJuRO7SQ FOHEarScWHgogMd0becdnOPBvmz2U+xLDoi7rQBRlsumlRD6/GipjqofOwzfIM7gzziT q4mKlPFAORySkjXii3n0cYyG1LYHJni0Bt741kYf80/LsV9X08uQlBNt5RDzfcB9TrFm TmSkbTegYl2FtA+NKjFpDWJX+00C8ov7YblroUZB4iFr9vFVyMxNOQcxMKOwzvmYGAPF 97jO/z8XBTIAmazvBCOviW0I52cC4WPo8fQtqXpl3e+01/4Ep5hira/mmwQQ9HX8ul/d u+Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=MgnsQNNGmrb1/WPFPmQ/KLD2qYdy4H6vKjniKjTVkwc=; b=RLffKGODx80kgJb4hrbuP6N/m5rvml6rie6n5nZHr8RmjMZ8dCG7Uwl5dhTy1vtmB+ mNzuXzsp5VdmQ6azbYEf7kz4IY5Z0igZAWL1g2X+WNAngV5h7xFj7cpP/lTSHkrXa5Q0 N1hv+PL9vNOoXvgHtZNPP4tYAsuUgG5vTCWkvnM2wg4nxH5+e6sfm58As06y/IQovoUl 7BBcMOapzz/WpEH0OLdFEyNUi3su5ZbyJ+tcAzhwRL66YawkSR8N5YHlLNBTfqHeOA8U f/sTosnlBDfdB5AiIxBk40Lv7DFXhLtz7iGhx6viJkvQhjplrM8VCsHssCu6akqmnY5a 6C9g== X-Gm-Message-State: AIkVDXLc+1G322qKQ5h8QffvIiRG8RGc3FNj8hV5M+4cLmp2iZEKrIslXKcR/vRZ9W9xa18OfTNOwJ/Z+vg84A== X-Received: by 10.31.73.66 with SMTP id w63mr23790207vka.106.1483545389953; Wed, 04 Jan 2017 07:56:29 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Alexander Dejanovski Date: Wed, 04 Jan 2017 15:56:19 +0000 Message-ID: Subject: Re: Reaper repair seems to "hang" To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a114dc2a4fa0e90054546d324 archived-at: Wed, 04 Jan 2017 15:56:43 -0000 --001a114dc2a4fa0e90054546d324 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Actually, the problem is related to CASSANDRA-11430 . Before 2.2.6, the notification service did not work with newly deprecated repair methods, on which Reaper still currently relies. C* 2.2.6 and onwards are not affected by this problem and work fine with Reaper. We're working on switching to the new repair method for 2.2 and 3.0/3.x, which should be ready in a few days/weeks. When using incremental repair, watch out for CASSANDRA-11696 which was fixed in C* 2.1.15, 2.2.7, 3.0.8 and 3.8. In prior versions, unrepaired SSTables can be marked as repaired, and thus never be repaired. Cheers, On Wed, Jan 4, 2017 at 6:09 AM Bhuvan Rawal wrote: > Hi Daniel, > > Looks like yours is a different case. If you're running incremental repai= r > for the first time it make take long time esp. if table is large. And > repair may seem to stuck even when things are working. > > You can try nodetool compactionstats when repair appears stuck, you'll > find a validation compaction happening if that's indeed the case. > > For the first incremental repair you can follow this doc, in further > repairs incremental repair should encounter very few sstables: > > https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepair= NodesMigration.html > > Regards, > Bhuvan > > > > On Jan 4, 2017 3:52 AM, "Daniel Kleviansky" wrote= : > > Hi Bhuvan, > > Thank you so very much for your detailed reply. > Just to ensure everyone is across the same information, and responses are > not duplicated across two different forums, I thought I'd share with the > mailing list that I've created a GitHub issue at: > https://github.com/thelastpickle/cassandra-reaper/issues/39 > > Kind regards, > Daniel > > On Wed, Jan 4, 2017 at 6:31 AM, Bhuvan Rawal wrote: > > Hi Daniel, > > We faced a similar issue during repair with reaper. We ran repair with > more repair threads than number of cassandra nodes. But on and off repair > was getting stuck and we had to do rolling restart of cluster or wait for > lock time to expire (~1hr). > > We had a look at the stuck repair, threadpools were getting stuck at > AntiEntropy stage. From the synchronized block in repair code it appeared > that per node at max 1 concurrent repair session per node is possible. > > According to > https://medium.com/@mlowicki/cassandra-reaper-introduction-ed73410492bf#.= f0erygqpk > : > > Segment runner has protection mechanism to avoid overloading nodes using > two simple rules to postpone repair if: > > 1. Number of pending compactions is greater than *MAX_PENDING_COMPACTIONS= * (20 > by default) > *2. Node is already running repair job* > > We tried running reaper with number of threads less than number of nodes > (assuming reaper will not submit multiple segments to single cassandra > node) but still it was observed that multiple repair segments were going = to > same node concurrently and threfore chances of nodes getting stuck in tha= t > state was possible. Finally we settled with single repair thread in reape= r > settings. Although takes a slightly more time but has completed > successfully numerous times. > > Thread Dump of cassandra server when repair was getting stuck: > > "*AntiEntropyStage:1" #159 daemon prio=3D5 os_prio=3D0 tid=3D0x00007f0fa1= 6226a0 > nid=3D0x3c82 waiting for monitor entry [0x00007ee9eabaf000*] > java.lang.Thread.State: BLOCKED (*on object monitor*) > at > org.apache.cassandra.service.ActiveRepairService.removeParentRepairSessio= n(ActiveRepairService.java:392) > - waiting to lock <0x000000067c083308> (a > org.apache.cassandra.service.ActiveRepairService) > at > org.apache.cassandra.service.ActiveRepairService.doAntiCompaction(ActiveR= epairService.java:417) > at org.apache.cassandra.repair > .RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:145) > at org.apache.cassandra.net > .MessageDeliveryTask.run(MessageDeliveryTask.java:67) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1142) > > Hope it helps! > > Regards, > Bhuvan > > According to > https://medium.com/@mlowicki/cassandra-reaper-introduction-ed73410492bf#.= f0erygqpk > : > > Segment runner has protection mechanism to avoid overloading nodes using > two simple rules to postpone repair if: > > 1. Number of pending compactions is greater than *MAX_PENDING_COMPACTIONS= * (20 > by default) > 2. Node is already running repair job > > > On Tue, Jan 3, 2017 at 11:16 AM, Alexander Dejanovski < > alex@thelastpickle.com> wrote: > > Hi Daniel, > > could you file a bug in the issue tracker ? > https://github.com/thelastpickle/cassandra-reaper/issues > > We'll figure out what's wrong and get your repairs running. > > Thanks ! > > On Tue, Jan 3, 2017 at 12:35 AM Daniel Kleviansky > wrote: > > Hi everyone, > > Using The Last Pickle's fork of Reaper, and unfortunately running into a > bit of an issue. I'll try break it down below. > > # Problem Description: > * After starting repair via the GUI, progress remains at 0/x. > * Cassandra nodes calculate their respective token ranges, and then > nothing happens. > * There were no errors in the Reaper or Cassandra logs. Only a message of > acknowledgement that a repair had initiated. > * Performing stack trace on the running JVM, once can see that the thread > spawning the repair process was waiting on a lock that was never being > released. > * This occurred on all nodes, and prevented any manually initiated repair > process from running. A rolling restart of each node was required, after > which one could run a `nodetool repair` successfully. > > # Cassandra Cluster Details: > * Cassandra 2.2.5 running on Windows Server 2008 R2 > * 6 node cluster, split across 2 DCs, with RF =3D 3:3. > > # Reaper Details: > * Reaper 0.3.3 running on Windows Server 2008 R2, utilising a PostgreSQL > database. > > ## Reaper settings: > * Parallism: DC-Aware > * Repair Intensity: 0.9 > * Incremental: true > > Don't want to swamp you with more details or unnecessary logs, especially > as I'd have to sanitize them before sending them out, so please let me kn= ow > if there is anything else I can provide, and I'll do my best to get it to > you. > > =E2=80=8BKind regards, > Daniel > > -- > ----------------- > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > > > > > > -- > Daniel Kleviansky > System Engineer & CX Consultant > M: +61 (0) 499 103 043 <+61%20499%20103%20043> | E: daniel@kleviansky.com > | W: http://danielkleviansky.com > > > -- ----------------- Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com --001a114dc2a4fa0e90054546d324 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Actually, the problem is related to=C2=A0CASSANDRA-11430.<= /span>

Before 2.2.6, the notification service did not work with newly depre= cated repair methods, on which Reaper still currently relies.
C* 2.2.6 a= nd onwards are not affected by this problem and work fine with Reaper.
<= br>We're working on switching to the new repair method for 2.2 and 3.0/= 3.x, which should be ready in a few days/weeks.

When usi= ng incremental repair, watch out for=C2=A0CASSANDRA-11696 which was fixed i= n C* 2.1.15, 2.2.7, 3.0.8 and 3.8. In prior versions, unrepaired SSTables c= an be marked as repaired, and thus never be repaired.
=C2=A0
Cheers= ,



On Wed, Jan 4, 2017 at 6:09 AM Bhuvan Rawal <bhu1rawal@gmail.com> wrote:
Hi = Daniel,

=
Looks like yours is a different case.= If you're running incremental repair for the first time it make take l= ong time esp. if table is large. And repair may seem to stuck even when thi= ngs are working.=C2=A0

You can try node= tool compactionstats when repair appears stuck, you'll find a validatio= n compaction happening if that's indeed the case.=C2=A0

For the first incremental repair you can follow this= doc, in further repairs incremental repair should encounter very few sstab= les:
https://docs.datastax.com/en/cassa= ndra/2.1/cassandra/operations/opsRepairNodesMigration.html

Regards,
Bhuvan

<= br class=3D"gmail_msg">
On Jan 4, 2017 3= :52 AM, "Daniel Kleviansky" <daniel@kleviansky.com>= wrote:
H= i Bhuvan,

Thank you so very much= for your detailed reply.
Just to ensure everyone is across the sa= me information, and responses are not duplicated across two different forum= s, I thought I'd share with the mailing list that I've created a Gi= tHub issue at:=C2=A0https://github.com/t= helastpickle/cassandra-reaper/issues/39

Kind regards,
Daniel

On Wed, Jan 4, 2017 at= 6:31 AM, Bhuvan Rawal <bhu1rawa= l@gmail.com> wrote:
Hi Daniel,
We faced a similar issue during repair with reaper. We ran repair with mor= e repair threads than number of cassandra nodes. But on and off repair was = getting stuck and we had to do rolling restart of cluster or wait for lock = time to expire (~1hr).=C2=A0

We had a look at the stuck repair, t= hreadpools were getting stuck at AntiEntropy stage. From the synchronized b= lock in repair code it appeared that per node at max 1 concurrent repair se= ssion per node is possible.=C2=A0

According to=C2=A0https://medium.= com/@mlowicki/cassandra-reaper-introduction-ed73410492bf#.f0erygqpk=C2=A0:
Segment runner has protection mechanism to a= void overloading nodes using two simple rules to postpone=C2=A0repair=C2=A0if:=C2=A0

1. Number of pending compactions is greater than=C2= =A0MAX_PENDING_COMPACTIONS=C2=A0(20 = by default)
2. Node is already runnin= g=C2=A0repair=C2=A0job

We tried running reaper with number of threads less than number of n= odes (assuming reaper will not submit multiple segments to single cassandra= node) but still it was observed that multiple repair segments were going t= o same node concurrently and threfore chances of nodes getting stuck in tha= t state was possible. Finally we settled with single repair thread in reape= r settings. Although takes a slightly more time but has completed successfu= lly numerous times.

<= /div>
Thread Dump of cassandra server when repair w= as getting stuck:

"AntiEntropyStage:1" #159 daemon prio=3D5 os_prio=3D0 ti= d=3D0x00007f0fa16226a0 nid=3D0x3c82 waiting for monitor entry [0x00007ee9ea= baf000]
=C2=A0 =C2=A0java.= lang.Thread.State: BLOCKED (on object monitor)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 a= t org.apache.cassandra.service.ActiveRepairService.removeParentRepairSessio= n(ActiveRepairService.java:392)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 - waiting to lock <0x000000067c083308> = (a org.apache.cassandra.service.ActiveRepairService)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.= service.ActiveRepairService.doAntiCompaction(ActiveRepairService.java:417)<= /font>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 a= t org.apache.cassandra.repair.RepairMessage= VerbHandler.doVerb(RepairMessageVerbHandler.java:145)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.ca= ssandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.= util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.ut= il.concurrent.FutureTask.run(FutureTask.java:266)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.util.concurrent.Thr= eadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
<= div class=3D"gmail_msg">
Hope it = helps!

Regards,
Bhuvan

According to=C2=A0http= s://medium.com/@mlowicki/cassandra-reaper-introduction-ed73410492bf#.f0eryg= qpk=C2=A0:<= /div>

Segment runner has protecti= on mechanism to avoid overloading nodes using two simple rules to postpone= =C2=A0repair=C2=A0if:=C2=A0
=
1. Number of pending compactions i= s greater than=C2=A0MAX_PENDING_COMPACTIONS=C2=A0(20 by default)
2. Node is already running=C2= =A0repair=C2=A0job


On Tue, Jan 3, 2017 at 11:16= AM, Alexander Dejanovski <al= ex@thelastpickle.com> wrote:
Hi Dan= iel,

could you file a bug in the issue tracker ?=C2=A0https://github.com/thelastpickle/cassandra-reaper/issues= =C2=A0

We'll figure out what's wrong and get your repairs= running.

Thanks !

On Tue, Jan 3, 2017 at 12:35 AM Daniel Klevian= sky <daniel@kleviansky.com> wrote:
Hi everyone,

Using The= Last Pickle's fork of Reaper, and unfortunately running into a bit of = an issue. I'll try break it down below.

# Problem Description:
* After starting repair via the GUI, progress remains at 0/x.
* Cassandra nodes calculate their respective to= ken ranges, and then nothing happens.
= * There were no errors in the Reaper or Cassandra logs. Only a message of = acknowledgement that a repair had initiated.
* Performing stack trace on the running JVM, once can see that the = thread spawning the repair process was waiting on a lock that was never bei= ng released.
* This occurred on all n= odes, and prevented any manually initiated repair process from running. A r= olling restart of each node was required, after which one could run a `node= tool repair` successfully.

= # Cassandra Cluster Details:
* Cassa= ndra 2.2.5 running on Windows Server 2008 R2
* 6 node cluster, split across 2 DCs, with RF =3D 3:3.
=

# Reaper Details:
* Reaper 0.3.3 running on Windows Server 2008 R2, utilisi= ng a PostgreSQL database.

= ## Reaper settings:
* Parallism: DC-= Aware
* Repair Intensity: 0.9<= /div>
* Incremental: true

=
Don't want to swamp you with more details or = unnecessary logs, especially as I'd have to sanitize them before sendin= g them out, so please let me know if there is anything else I can provide, = and I'll do my best to get it to you.

=E2=80=8BKind regards,
Daniel
--
-----------------
Alexander Dejanovski
F= rance
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

=



--
Daniel Kleviansky
System Eng= ineer & CX Consultant
M: = +61 (0) 499 103 043 | E: daniel@kleviansky.= com | W:=C2=A0http://danielkleviansky.com<= /div>
<= div dir=3D"ltr">--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache = Cassandra Consulting
<= /div>
--001a114dc2a4fa0e90054546d324--