Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
Date: Wed, 27 Jun 2012 17:49:48 +0200
Message-ID: 
 <CADVHTB-kxEWNHgE2HY0C=e=RN+gLPFA73tnabYgRgq3PL4dfgQ@mail.gmail.com>
Subject: Node crashing during read repair
From: Robin Verlangen <robin@us2.nl>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=20cf3079bdda376a3504c3762a27

--20cf3079bdda376a3504c3762a27
Content-Type: text/plain; charset=ISO-8859-1

Hi there,

Today I found one node (running 1.1.1 in a 3 node cluster) being dead for
the third time this week, it died with the following message:

ERROR [ReadRepairStage:3] 2012-06-27 14:28:30,929
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[ReadRepairStage:3,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
        at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
        at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
        at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:566)
        at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:439)
        at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:391)
        at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:372)
        at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:460)
        at
org.apache.cassandra.service.RowRepairResolver.scheduleRepairs(RowRepairResolver.java:136)
        at
org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:94)
        at
org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)

Is this a common bug in 1.1.1, or did I get a "race" condition?

-- 
With kind regards,

Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

--20cf3079bdda376a3504c3762a27
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi there,<div><br></div><div>Today I found one node (running 1.1.1 in a 3 n=
ode cluster) being dead for the third time this week, it died with the foll=
owing message:</div><div><br></div><div><div>ERROR [ReadRepairStage:3] 2012=
-06-27 14:28:30,929 AbstractCassandraDaemon.java (line 134) Exception in th=
read Thread[ReadRepairStage:3,5,main]</div>
<div>java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor ha=
s shut down</div><div>=A0 =A0 =A0 =A0 at org.apache.cassandra.concurrent.De=
buggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor=
.java:60)</div>
<div>=A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.reject(Unkn=
own Source)</div><div>=A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExe=
cutor.execute(Unknown Source)</div><div>=A0 =A0 =A0 =A0 at org.apache.cassa=
ndra.net.MessagingService.receive(MessagingService.java:566)</div>
<div>=A0 =A0 =A0 =A0 at org.apache.cassandra.net.MessagingService.sendOneWa=
y(MessagingService.java:439)</div><div>=A0 =A0 =A0 =A0 at org.apache.cassan=
dra.net.MessagingService.sendRR(MessagingService.java:391)</div><div>=A0 =
=A0 =A0 =A0 at org.apache.cassandra.net.MessagingService.sendRR(MessagingSe=
rvice.java:372)</div>
<div>=A0 =A0 =A0 =A0 at org.apache.cassandra.net.MessagingService.sendRR(Me=
ssagingService.java:460)</div><div>=A0 =A0 =A0 =A0 at org.apache.cassandra.=
service.RowRepairResolver.scheduleRepairs(RowRepairResolver.java:136)</div>=
<div>=A0 =A0 =A0 =A0 at org.apache.cassandra.service.RowRepairResolver.reso=
lve(RowRepairResolver.java:94)</div>
<div>=A0 =A0 =A0 =A0 at org.apache.cassandra.service.AsyncRepairCallback$1.=
runMayThrow(AsyncRepairCallback.java:54)</div><div>=A0 =A0 =A0 =A0 at org.a=
pache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)</div><di=
v>=A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask=
(Unknown Source)</div>
<div>=A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(=
Unknown Source)</div><div>=A0 =A0 =A0 =A0 at java.lang.Thread.run(Unknown S=
ource)</div><div><br></div><div>Is this a common bug in 1.1.1, or did I get=
 a &quot;race&quot; condition?</div>
<div><br></div>-- <br>With kind regards,<div><br></div><div>Robin Verlangen=
</div><div><i>Software engineer</i></div><div><i><br></i></div><div>W <a hr=
ef=3D"http://www.robinverlangen.nl" target=3D"_blank">http://www.robinverla=
ngen.nl</a></div>
<div>E <a href=3D"mailto:robin@us2.nl" target=3D"_blank">robin@us2.nl</a></=
div><div><br></div><div><font color=3D"#666666">Disclaimer: The information=
 contained in this message and attachments is intended solely for the atten=
tion and use of the named addressee and may be confidential. If you are not=
 the intended recipient, you are reminded that the information remains the =
property of the sender. You must not use, disclose, distribute, copy, print=
 or rely on this e-mail. If you have received this message in error, please=
 contact the sender immediately and irrevocably delete this message and any=
 copies.</font></div>
<br>
</div>

--20cf3079bdda376a3504c3762a27--