Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
From: Aaron Morton <aaron@thelastpickle.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_1EEB67BF-1842-46D8-A6B6-ABE8308E54EF"
Message-Id: <9839670A-E34B-4D1D-A6DA-D191C10B58FC@thelastpickle.com>
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: HintedHandoff process does not finish
Date: Mon, 30 Sep 2013 20:44:08 +1300
References: 
 <CAADG=Lh7BmKxrzQeZUkTonVT9FX6Bqt95zSuG+Ly6yWsR_MM7Q@mail.gmail.com>
To: user@cassandra.apache.org
In-Reply-To: 
 <CAADG=Lh7BmKxrzQeZUkTonVT9FX6Bqt95zSuG+Ly6yWsR_MM7Q@mail.gmail.com>


--Apple-Mail=_1EEB67BF-1842-46D8-A6B6-ABE8308E54EF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

> What can be the reason for the handoff process not to finish?
Check for other errors about timing out during hint reply.=20

> What would be the best way to recover from this situation?
If they are really causing trouble drop the hints via =
HintedHandoffManager JMX MBean or stopping the node and deleting the =
files on disk. Then use repair later.=20

> What can be done to prevent this from happening again?
Hints are stored when either the node is down before the request starts =
or when the coordinator times out waiting for the remote node. Check the =
logs for nodes going down, and check the MessagingService MBean for =
TimedOuts from other nodes. This may indicate issues with a cross DC =
connection.=20

Cheers

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 27/09/2013, at 11:18 PM, Tom van den Berge <tom@drillster.com> wrote:

> Hi,
>=20
> One one of my nodes, the (storage) load increased dramatically =
(doubled), within one or two hours. The hints column family was causing =
the growth. I noticed one HintedHandoff process that was started some =
two hours ago, but hadn't finished. Normally, these processes take only =
a few seconds, 15 seconds max, in my cluster.
>=20
> The not-finishing process was handing the hints over to a host in =
another data center. There were no warning or error messages in the =
logs, other than the repeated "flushing high-traffic column family =
hints".
> I'm using Cassandra 1.2.3.
> What can be the reason for the handoff process not to finish?
> What would be the best way to recover from this situation?
> What can be done to prevent this from happening again?
>=20
> Thanks in advance,
> Tom
>=20
>=20
>=20
>=20
>=20


--Apple-Mail=_1EEB67BF-1842-46D8-A6B6-ABE8308E54EF
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=iso-8859-1

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Diso-8859-1"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><blockquote type=3D"cite"><div dir=3D"ltr"><ul><li>What can be the =
reason for the handoff process not to =
finish?<br></li><li></li></ul></div></blockquote>Check for other errors =
about timing out during hint reply.&nbsp;<div><br></div><div><blockquote =
type=3D"cite"><div dir=3D"ltr"><ul><li>What would be the best way to =
recover from this situation?</li><li></li></ul></div></blockquote>If =
they are really causing trouble drop the hints via HintedHandoffManager =
JMX MBean or stopping the node and deleting the files on disk. Then use =
repair later.&nbsp;</div><div><br></div><div><blockquote =
type=3D"cite"><div dir=3D"ltr"><ul><li>What can be done to prevent this =
from happening again?</li><li></li></ul></div></blockquote>Hints are =
stored when either the node is down before the request starts or when =
the coordinator times out waiting for the remote node. Check the logs =
for nodes going down, and check the MessagingService MBean for TimedOuts =
from other nodes. This may indicate issues with a cross DC =
connection.&nbsp;</div><div><br></div><div>Cheers</div><div><br><div =
apple-content-edited=3D"true">
<div style=3D"color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
medium; font-style: normal; font-variant: normal; font-weight: normal; =
letter-spacing: normal; line-height: normal; orphans: 2; text-align: =
-webkit-auto; text-indent: 0px; text-transform: none; white-space: =
normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div>-----------------</div><div>Aaron Morton</div><div>New =
Zealand</div><div>@aaronmorton</div><div><br></div><div>Co-Founder &amp; =
Principal Consultant</div><div>Apache Cassandra Consulting</div><div><a =
href=3D"http://www.thelastpickle.com">http://www.thelastpickle.com</a></di=
v></div>
</div>
<br><div><div>On 27/09/2013, at 11:18 PM, Tom van den Berge &lt;<a =
href=3D"mailto:tom@drillster.com">tom@drillster.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div dir=3D"ltr"><div>Hi,</div><div><br></div><div>One one =
of my nodes, the (storage) load increased dramatically (doubled), within =
one or two hours. The hints column family was causing the growth. I =
noticed one HintedHandoff process that was started some two hours ago, =
but hadn't finished. Normally, these processes take only a few seconds, =
15 seconds max, in my cluster.</div>
<div><br></div><div>The not-finishing process was handing the hints over =
to a host in another data center. There were no warning or error =
messages in the logs, other than the repeated "flushing high-traffic =
column family hints".</div>
<div>I'm using Cassandra 1.2.3.</div><div><ul><li>What can be the reason =
for the handoff process not to finish?<br></li><li>What would be the =
best way to recover from this situation?</li><li>What can be done to =
prevent this from happening again?</li>
</ul><div><br></div></div><div>Thanks in =
advance,</div><div>Tom</div><div><br></div><div><span =
style=3D"color:rgb(119,119,119);font-family:'normal =
arial',sans-serif;font-size:12px"><br></span></div><div><br></div>
<br><div><br class=3D"webkit-block-placeholder"></div>
</div>
</blockquote></div><br></div></body></html>=

--Apple-Mail=_1EEB67BF-1842-46D8-A6B6-ABE8308E54EF--