Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
From: =?utf-8?Q?Hannu_Kr=C3=B6ger?= <hkroger@gmail.com>
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\))
Subject: Repair fails for unknown reason
Message-Id: <CEDA8884-1691-4203-B5E0-8E3D56C791A1@gmail.com>
Date: Wed, 3 Jan 2018 18:23:11 +0200
To: user <user@cassandra.apache.org>
archived-at: Wed, 03 Jan 2018 16:23:24 -0000

Hello,

Situation is as follows:

Repair was started on node X on this keyspace with =E2=80=94full =E2=80=94=
pr. Repair fails on node Y.

Node Y has debug logging on (DEBUG on org.apache.cassandra) and I=E2=80=99=
m looking at the debug.log. I see following messages related to this =
repair request:

-----------
DEBUG [AntiEntropyStage:1] 2018-01-02 17:52:12,530 =
RepairMessageVerbHandler.java:114 - Validating =
ValidationRequest{gcBefore=3D1511473932} =
org.apache.cassandra.repair.messages.ValidationRequest@5a17430c
DEBUG [ValidationExecutor:4] 2018-01-02 17:52:12,531 =
StorageService.java:3321 - Forcing flush on keyspace mykeyspace, CF =
mytable
DEBUG [MemtablePostFlush:54] 2018-01-02 17:52:12,531 =
ColumnFamilyStore.java:954 - forceFlush requested but everything is =
clean in mytable
ERROR [ValidationExecutor:4] 2018-01-02 17:52:12,532 Validator.java:268 =
- Failed creating a merkle tree for [repair =
#1df000a0-effa-11e7-8361-b7c9edfbfc33 on mykeyspace/mytable, =
[(6917529027641081856,-9223372036854775808]]], /123.123.123.123 (see log =
for details)
-----------

then the same about another table and after that which indicates that =
repair =E2=80=9Cmaster=E2=80=9D has told to abort basically, right?

-----------
DEBUG [AntiEntropyStage:1] 2018-01-02 17:52:12,563 =
RepairMessageVerbHandler.java:142 - Got anticompaction request =
AnticompactionRequest{parentRepairSession=3D1de949e0-effa-11e7-8361-b7c9ed=
fbfc33} =
org.apache.cassandra.repair.messages.AnticompactionRequest@5dc8be
ea
ERROR [AntiEntropyStage:1] 2018-01-02 17:52:12,563 =
RepairMessageVerbHandler.java:168 - Got error, removing parent repair =
session
ERROR [AntiEntropyStage:1] 2018-01-02 17:52:12,564 =
CassandraDaemon.java:228 - Exception in thread =
Thread[AntiEntropyStage:1,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: Parent repair =
session with id =3D 1de949e0-effa-11e7-8361-b7c9edfbfc33 has failed.
        at =
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageV=
erbHandler.java:171) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at =
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:=
66) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at =
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) =
~[na:1.8.0_111]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) =
~[na:1.8.0_111]
        at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1142) ~[na:1.8.0_111]
        at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:617) [na:1.8.0_111]
        at =
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeall=
ocator$0(NamedThreadFactory.java:81) =
[apache-cassandra-3.11.0.jar:3.11.0]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
Caused by: java.lang.RuntimeException: Parent repair session with id =3D =
1de949e0-effa-11e7-8361-b7c9edfbfc33 has failed.
        at =
org.apache.cassandra.service.ActiveRepairService.getParentRepairSession(Ac=
tiveRepairService.java:409) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at =
org.apache.cassandra.service.ActiveRepairService.doAntiCompaction(ActiveRe=
pairService.java:444) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at =
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageV=
erbHandler.java:143) ~[apache-cassandra-3.11.0.jar:3.11.0]
        ... 7 common frames omitted
-----------

But that is almost all in the log and I don=E2=80=99t really see what =
the original problem here is.=20

Cassandra flushes the table to start building merkle tree and on next =
millisecond it already fails the repair but without proper exception or =
error logging about the problem.

Cassandra version is the 3.11.0.

Any ideas?

Cheers,
Hannu=

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org