Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of tmarthinussen@gmail.com
 designates 209.85.210.172 as permitted sender)
References: <4ED742D7.4040108@sitevision.se>
 <CAKkz8Q3H0KVGDVc_c56xBPcVmuB7Gs30nn4k1aurvs5OjfRa_w@mail.gmail.com>
In-Reply-To: 
 <CAKkz8Q3H0KVGDVc_c56xBPcVmuB7Gs30nn4k1aurvs5OjfRa_w@mail.gmail.com>
Mime-Version: 1.0 (1.0)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8
Message-Id: <1F11BA02-CAEB-4DEF-969A-DB0A439630AB@gmail.com>
Cc: "user@cassandra.apache.org" <user@cassandra.apache.org>
From: Terje Marthinussen <tmarthinussen@gmail.com>
Subject: Re: Hinted handoff bug?
Date: Fri, 2 Dec 2011 08:59:03 +0900
To: "user@cassandra.apache.org" <user@cassandra.apache.org>

Sorry for not checking source to see if things have changed but i just remem=
bered an issue I have forgotten to make jira for.

In old days, nodes would periodically try to deliver queues.

However, this was at some stage changed so it only deliver if a node is bein=
g marked up.

However, you can definitely have a scenario where  A fails to deliver to B s=
o it send the hint to C instead.

However, B is not really down, it just could not accept that packet at that t=
ime and C always (correctly in this case) thinks B is up and it never tries t=
o deliver the hints to B.

Will this change fix this, or do we need to get back the thread that periodi=
cally tried to deliver hints regardless of node status changes?

Regards,
Terje

On 1 Dec 2011, at 19:10, Sylvain Lebresne <sylvain@datastax.com> wrote:

> You're right, good catch.
> Do you mind opening a ticket on jira
> (https://issues.apache.org/jira/browse/CASSANDRA)?
>=20
> --
> Sylvain
>=20
> On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigb=C3=A4ck
> <fredrik.l.stigback@sitevision.se> wrote:
>> Hi,
>> We,re running cassandra 1.0.3.
>> I've done some testing with 2 nodes (node A, node B), replication factor 2=
.
>> I take node A down, writing some data to node B and then take node A up.
>> Sometimes hints aren't delivered when node A comes up.
>>=20
>> I've done some debugging in org.apache.cassandra.db.HintedHandOffManager a=
nd
>> sometimes node B ends up in a strange state in method
>> org.apache.cassandra.db.HintedHandOffManager.deliverHints(final InetAddre=
ss
>> to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries
>> already has node A in it's Set and therefore no hints will ever be delive=
red
>> to node A.
>> The only reason for this that I can see is that in
>> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetA=
ddress
>> endpoint) the hintStore.isEmpty() check returns true and the endpoint (no=
de
>> A)  isn't removed from
>> org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hi=
nts
>> will ever be delivered again until node B is restarted.
>> During what conditions will hintStore.isEmpty() return true?
>> Shouldn't the hintStore.isEmpty() check be inside the try {} finally{}
>> clause, removing the endpoint from queuedDeliveries in the finally block?=

>>=20
>> public void deliverHints(final InetAddress to)
>> {
>>         logger_.debug("deliverHints to {}", to);
>>         if (!queuedDeliveries.add(to))
>>             return;
>>         .......
>> }
>>=20
>> private void deliverHintsToEndpoint(InetAddress endpoint) throws
>> IOException, DigestMismatchException, InvalidRequestException,
>> TimeoutException,
>> {
>>         ColumnFamilyStore hintStore =3D
>> Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
>>         if (hintStore.isEmpty())
>>             return; // nothing to do, don't confuse users by logging a no=
-op
>> handoff
>>     try
>>     {
>>         ......
>>     }
>>     finally
>>     {
>>             queuedDeliveries.remove(endpoint);
>>     }
>> }
>>=20
>> Regards
>> /Fredrik