Hi,

HintsColumnFamily directory on Node 1 (the first to be upgraded):
1.8K Oct 27 11:27 system-HintsColumnFamily-hf-2-Data.db
79 Oct 27 11:27 system-HintsColumnFamily-hf-2-Digest.sha1
496 Oct 27 11:27 system-HintsColumnFamily-hf-2-Filter.db
26 Oct 27 11:27 system-HintsColumnFamily-hf-2-Index.db
4.3K Oct 27 11:27 system-HintsColumnFamily-hf-2-Statistics.db


HintsColumnFamily-directory on Node 2 (the second to be upgraded):
12K Oct 27 11:37 system-HintsColumnFamily-hf-3-Data.db
79 Oct 27 11:37 system-HintsColumnFamily-hf-3-Digest.sha1
976 Oct 27 11:37 system-HintsColumnFamily-hf-3-Filter.db
11 Oct 27 11:37 system-HintsColumnFamily-hf-3-Index.db
43K Oct 27 11:37 system-HintsColumnFamily-hf-3-Statistics.db

11:37 is exactly ten minutes after the node was started after upgrading, according to the logs, which means that the first node's sstables is generated when the second node starts.



The sstable2json is more or less equal for both nodes, but node 1 starts with "40000000000000000000000000000000" and node 2 with "00".
The rest looks like this:
"00": {"f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX": {"deletedAt": -9223372036854775808, "subColumns": []}, "f7XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX": {"deletedAt": -9223372036854775808, "subColumns": []},
(The f7XXXXXXXXXXXXXXXXX is repeated with multiple different keys, but all are above f7XXXXXXXXXXXXXXXXXXXXX.  The deletedAt is always equal - on both nodes)

Of course, this could be caused by me doing a mistake while upgrading.  I have not tried to compact, but repair runs every night.  It's a low traffic cluster to test cassandra, if that matters.

.vegard,



----- Original Message -----
From:
user@cassandra.apache.org

To:
<user@cassandra.apache.org>
Cc:

Sent:
Sun, 11 Nov 2012 13:16:01 -0600
Subject:
Re: Hinted Handoff runs every ten minutes


How many hint sstables are there? What does sstable2json show?

On Thu, Nov 8, 2012 at 3:23 PM, Mike Heffner <mike@librato.com> wrote:
> Is there a ticket open for this for 1.1.6?
>
> We also noticed this after upgrading from 1.1.3 to 1.1.6. Every node runs a
> 0 row hinted handoff every 10 minutes. N-1 nodes hint to the same node,
> while that node hints to another node.
>
>
> On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget <post@fantasista.no> wrote:
>>
>> Hi,
>>
>> I have the exact same problem with 11.6. HintsColumnFamily consists of
>> one row (Rowkey 00, nothing more). The "problem" started after upgrading
>> from 1.1.4 to 1.1.6. Every ten minutes HintedHandoffManager starts and
>> finishes after sending "0 rows".
>>
>> .vegard,
>>
>>
>>
>> ----- Original Message -----
>> From:
>> user@cassandra.apache.org
>>
>> To:
>> <user@cassandra.apache.org>
>> Cc:
>>
>> Sent:
>> Mon, 29 Oct 2012 23:45:30 +0100
>>
>> Subject:
>> Re: Hinted Handoff runs every ten minutes
>>
>>
>> Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
>> > I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
>> >
>> > How can I check to see why it keeps running HintedHandoff?
>> you have tombstone is system.HintsColumnFamily use list command in
>> cassandra-cli to check
>>
>
>
>
> --
>
> Mike Heffner <mike@librato.com>
> Librato, Inc.
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com