incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radim Kolar <...@sendmail.cz>
Subject hintedhandoff in 1.0.3
Date Tue, 15 Nov 2011 09:31:38 GMT

> I suspect these partial/invalid hints are left over from a failed 
> hints delivery from before you upgraded to 1.0.3 and not something 
> created by 1.0.3.
> Try to clear HintsColumnFamily (by removing the sstables for example) 
> first and then see if you still can reproduce this issue afterwards.
it still does not work, but there is an improvement over 1.0.2. 
currently it looks like this:

  INFO [HintedHandoff:1] 2011-11-15 06:42:59,794 
HintedHandOffManager.java (line 334) Finished hinted handoff of 0 rows 
to endpoint /

but there is lot of data not delivered. forced compaction shows it:

112) Compacting 
[SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-hb-63-Data.db'), 
SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily-hb-61-Data.db')]
  INFO [CompactionExecutor:41] 2011-11-15 07:08:24,034 
CompactionController.java (line 129) Compacting large row 
system/HintsColumnFamily:0758480e325aa09951c555e1249a69ba (237411479 
bytes) incrementally
  INFO [CompactionExecutor:41] 2011-11-15 07:08:33,418 
CompactionTask.java (line 213) Compacted to 
[/var/lib/cassandra/data/system/HintsColumnFamily-hb-64-Data.db,].  
237,411,643 to 237,411,505 (~99% of original) bytes for 1 keys at 
24.091642MB/s.  Time: 9,398ms.

after node restart it still shows 0 rows delivered. hints should be 
newly written by 1.0.3. i deleted old rows from older versions in hints 
- only 2 tombstones left there before another batch of hints testing.

hints inside are not from supercolumn CF. I remember to have 2 rows in 
HintsColumnFamily, one was successfully delivered and this one seems to 
be stuck. These hints are older than 1 hour.

Did you tested outgoing hints delivery with more than row there? its 
probably 1 key = 1 hints delivery session and if node is back up for 
while then second hints session is created. Try to do your testing with 
about 500k hints with multiple sessions.

Mime
View raw message