Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of
 prvs=39163d4bb=bburruss@expedia.com designates 216.251.112.221 as permitted
 sender)
From: Todd Burruss <bburruss@expedia.com>
To: "dev@cassandra.apache.org" <dev@cassandra.apache.org>
Date: Mon, 20 Feb 2012 21:10:05 -0800
Subject: Re: hinted handoff - 1.1.0 beta1
Thread-Topic: hinted handoff - 1.1.0 beta1
Thread-Index: AczwVwustQa7ab6ZS1SoNF+XhSlZjQ==
Message-ID: <CB6864E9.486B%bburruss@expedia.com>
In-Reply-To: 
 <CALdd-zgy0h0zZFSzE+cYwRrgqGPVeywshYKGZVC1HqZ78k53Ug@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.14.0.111121
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://issues.apache.org/jira/browse/CASSANDRA-3935


On 2/20/12 7:07 PM, "Jonathan Ellis" <jbellis@gmail.com> wrote:

>Hmm, that dates back to early 0.7...  I think the intent is to skip
>deleted hints, even if the tombstoned hints haven't expired yet (and
>thus would not be elided by the getCF filter).  Sounds like that's
>breaking pagination.  Normally, all hints get delivered at once and
>then we compact away the tombstones -- that's my guess as to why we
>haven't hit this before.
>
>Can you create a ticket?
>
>On Mon, Feb 20, 2012 at 4:52 PM, Todd Burruss <bburruss@expedia.com>
>wrote:
>> I'm testing hinted handoff in 1.1 beta1 and cannot seem to get a hint
>>delivered.  3 node cluster, RF =3D 3, writing with CL =3D ONE.  killed a
>>host then did the write using the CLI on another node.  I can see hint
>>waiting using CLI and I see the log messages at the end of this email.
>>this suggests the hints exist bu are not being delivered (and I'll see
>>the log messages over and over.)
>>
>> I did see tracing with debugger and see that in
>>HintedHandoffManager.deliverHintsToEndpointInternal, this line will
>>remove the hint because of the Integer.MAX_VALUE
>>
>>
>>            ColumnFamily hintsPage =3D
>>ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter),
>>Integer.MAX_VALUE);
>>
>> I'm not sure I quite understand while MAX is used when the same
>>"remove" is done in getColumnFamily(filter).  regardless if it is useful
>>or not, it prevents the hints from delivery.
>>
>> any thoughts?
>>
>>
>> [default@unknown] use system;
>> Authenticated to keyspace: system
>> [default@system] list hintscolumnfamily;
>> Using default limit of 100
>> -------------------
>> RowKey: 00
>> =3D> (super_column=3D493ecfa05c1411e100000da23097c7ff,
>>     (column=3D6b6579, value=3D6b35, timestamp=3D1329777799580, ttl=3D864=
00)
>>     (column=3D6d75746174696f6e,
>>value=3D000662746f64646200026b3500000001000003e801000003e87fffffff8000000=
00
>>00000000000000100026335000004b96d055fd1300000000c68696e746564207772697465
>>, timestamp=3D1329777799579, ttl=3D86400)
>>     (column=3D7461626c65, value=3D62746f646462, timestamp=3D132977779958=
0,
>>ttl=3D86400)
>>     (column=3D76657273696f6e, value=3D00000004, timestamp=3D132977779958=
0,
>>ttl=3D86400))
>>
>> 1 Row Returned.
>> Elapsed time: 58 msec(s).
>>
>>
>>  INFO [HintedHandoff:1] 2012-02-20 14:44:53,811
>>HintedHandOffManager.java (line 296) Started hinted handoff for token: 0
>>with IP: /192.168.56.1
>>  INFO [HintedHandoff:1] 2012-02-20 14:44:53,815
>>HintedHandOffManager.java (line 373) Finished hinted handoff of 0 rows
>>to endpoint /192.168.56.1
>>
>>
>> I have a hint waiting to be deli
>
>
>
>--=20
>Jonathan Ellis
>Project Chair, Apache Cassandra
>co-founder of DataStax, the source for professional Cassandra support
>http://www.datastax.com