cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yang Yang (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-3385) NPE in hinted handoff
Date Wed, 19 Oct 2011 19:49:10 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yang Yang updated CASSANDRA-3385:
---------------------------------

    Priority: Critical  (was: Major)

I just found that this contributes to another symptom I'm seeing: for RF=3, and a ring of
3 nodes, if I bring down 1 box, the remaining 2 still work fine for Quorum access, but the
latency is 20x high.

I can see from debugging that a lot of time is spent on storing hints into local system table
on the coordinator. but this Table.apply is slow because a lot of time is spent on the lock,
while it really should not happen since the lock is sharded into 4096 ones. it turns out that
all the keys used in the hints writing are the same key, at least in the examples I looked
at in the debugger, if I'm correct in this observation, this is a serious bug
                
> NPE in hinted handoff
> ---------------------
>
>                 Key: CASSANDRA-3385
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3385
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Yang Yang
>            Priority: Critical
>
> I'm using the current HEAD of 1.0.0 github branch, and I'm still seeing this error, not
sure if it's  this bug or another one.
>  INFO [HintedHandoff:1] 2011-10-19 12:43:17,674 HintedHandOffManager.java (line 263)
Started hinted handoff for token: 11342745564
> 0312821154458202477256070484 with IP: /10.39.85.140
> ERROR [HintedHandoff:1] 2011-10-19 12:43:17,885 AbstractCassandraDaemon.java (line 133)
Fatal exception in thread Thread[HintedHan
> doff:1,1,main]
> java.lang.RuntimeException: java.lang.NullPointerException
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:289)
>         at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
>         at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:337)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         ... 3 more
> ERROR [HintedHandoff:1] 2011-10-19 12:43:17,886 AbstractCassandraDaemon.java (line 133)
Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.NullPointerException
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
>         at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:289)
>         at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81)
>         at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:337)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         ... 3 more
> this could possibly be related to #3291

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message