cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jaakko Laine (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-634) Hinted Handoff Exception
Date Mon, 21 Dec 2009 09:54:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793134#action_12793134
] 

Jaakko Laine commented on CASSANDRA-634:
----------------------------------------

It was supposed to solve it, but obviously it did not fully do so.

Problem in your case might be because hinted handoff data is persistent and gossiper data
is not. Suppose there are nodes A and B. Suppose B goes down and A stores hinted data for
it. Later A is restarted -> A still has hinted data for B, but after restart its gossiper
knows nothing about B. It does not help even if we gossip about dead nodes, as nobody has
ever heard of B. If B is gone forever, A can never get rid of hinted data.

Don't know what would be the best thing to do here. removetoken command could make efforts
to redirect hints to new destination in case a hinted target is removed. However, if the endpoint
has been lost from gossip/tokenmetadata, then there is nothing it can do as it does not know
who the endpoint was. Another option would be to add manual command to redirect hinted data.

Other options?


> Hinted Handoff Exception
> ------------------------
>
>                 Key: CASSANDRA-634
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-634
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5
>            Reporter: Chris Goffinet
>            Assignee: Jaakko Laine
>             Fix For: 0.5
>
>         Attachments: 634-1st-part-gossip-about-all-nodes.patch
>
>
> Updated to the latest codebase from cassandra-0.5 branch. All nodes booted up fine and
then I start noticing this error:
> ERROR [HINTED-HANDOFF-POOL:1] 2009-12-14 22:05:34,191 CassandraDaemon.java (line 71)
Fatal exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
>         at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:253)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.NullPointerException
>         at org.apache.cassandra.gms.FailureDetector.isAlive(FailureDetector.java:146)
>         at org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:106)
>         at org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:177)
>         at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:75)
>         at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:249)
>         ... 3 more

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message