Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAEWpZH3wPLFh-HF1MC6FrhgAkvsaRkaBg3nUdExuWm73KZO_OA@mail.gmail.com>
References: 
 <CAEWpZH3wPLFh-HF1MC6FrhgAkvsaRkaBg3nUdExuWm73KZO_OA@mail.gmail.com>
Date: Thu, 14 Apr 2016 16:28:26 -0700
Message-ID: 
 <CAEWpZH2StEnQ_pC8K2EsPkfS9gXj2+r7-4T_hQyMS+46ECS+uQ@mail.gmail.com>
Subject: Fwd: Cassandra Load spike
From: kavya <kavyabharadwaj@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a113ff5a249108605307a40a3

--001a113ff5a249108605307a40a3
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

We are running a 6 node cassandra 2.2.4 cluster and we are seeing a spike
in the disk Load as per the =E2=80=98nodetool status=E2=80=99 command that =
does not
correspond with the actual disk usage. Load reported by nodetool was as
high as 3 times actual disk usage on certain nodes.
We noticed that the periodic repair failed with below error on running the
command : =E2=80=99nodetool repair -pr=E2=80=99

ERROR [RepairJobTask:2] 2016-04-12 15:46:29,902 RepairRunnable.java:243 -
Repair session 64b54d50-0100-11e6-b46e-a511fd37b526 for range
(-3814318684016904396,-3810689996127667017] failed with error [=E2=80=A6.]
Validation failed in /<ip>
org.apache.cassandra.exceptions.RepairException: [=E2=80=A6.] Validation fa=
iled in
<ip>
    at
org.apache.cassandra.repair.ValidationTask.treeReceived(ValidationTask.java=
:64)
~[apache-cassandra-2.2.4.jar:2.2.4]
    at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.=
java:183)
~[apache-cassandra-2.2.4.jar:2.2.4]
    at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepair=
Service.java:410)
~[apache-cassandra-2.2.4.jar:2.2.4]
    at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVe=
rbHandler.java:163)
~[apache-cassandra-2.2.4.jar:2.2.4]
    at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:6=
7)
~[apache-cassandra-2.2.4.jar:2.2.4]
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1=
142)
[na:1.8.0_40]
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:=
617)
[na:1.8.0_40]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40

We restarted all nodes in the cluster and ran a full repair which completed
successfully without any validation errors, however we still see Load spike
on the same nodes after a while. Please advice.

Thanks!

--001a113ff5a249108605307a40a3
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<br><div class=3D"gmail_quote"><div dir=3D"ltr"><div><b=
r>We are running a 6 node cassandra 2.2.4 cluster and we are seeing a spike=
 in the disk Load as per the =E2=80=98nodetool status=E2=80=99 command that=
 does not correspond with the actual disk usage. Load reported by nodetool =
was as high as 3 times actual disk usage on certain nodes.<br>We noticed th=
at the periodic repair failed with below error on running the command : =E2=
=80=99nodetool repair -pr=E2=80=99<br><br>ERROR [RepairJobTask:2] 2016-04-1=
2 15:46:29,902 RepairRunnable.java:243 - Repair session 64b54d50-0100-11e6-=
b46e-a511fd37b526 for range (-3814318684016904396,-3810689996127667017] fai=
led with error [=E2=80=A6.] Validation failed in /&lt;ip&gt;<br>org.apache.=
cassandra.exceptions.RepairException: [=E2=80=A6.] Validation failed in &lt=
;ip&gt;<br>=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.repair.ValidationTask=
.treeReceived(ValidationTask.java:64) ~[apache-cassandra-2.2.4.jar:2.2.4]<b=
r>=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.repair.RepairSession.validatio=
nComplete(RepairSession.java:183) ~[apache-cassandra-2.2.4.jar:2.2.4]<br>=
=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.ActiveRepairService.hand=
leMessage(ActiveRepairService.java:410) ~[apache-cassandra-2.2.4.jar:2.2.4]=
<br>=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.repair.RepairMessageVerbHand=
ler.doVerb(RepairMessageVerbHandler.java:163) ~[apache-cassandra-2.2.4.jar:=
2.2.4]<br>=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.net.MessageDeliveryTas=
k.run(MessageDeliveryTask.java:67) ~[apache-cassandra-2.2.4.jar:2.2.4]<br>=
=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(Thr=
eadPoolExecutor.java:1142) [na:1.8.0_40]<br>=C2=A0=C2=A0=C2=A0 at java.util=
.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:=
1.8.0_40]<br>=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:745) [n=
a:1.8.0_40 <br><br>We restarted all nodes in the cluster and ran a full rep=
air which completed successfully without any validation errors, however we =
still see Load spike on the same nodes after a while. Please advice.<br><br=
></div>Thanks!<br></div>
</div><br></div>

--001a113ff5a249108605307a40a3--