cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3681) Multiple threads can attempt hint handoff to the same target
Date Wed, 04 Jan 2012 15:49:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179576#comment-13179576
] 

Brandon Williams commented on CASSANDRA-3681:
---------------------------------------------

The problem here is that now the schema check comes before the FD check:

{noformat}
ERROR 15:45:37,154 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: Didin't receive gossiped schema from /10.179.64.227 in 60000ms
        at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:210)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:239)
        at org.apache.cassandra.db.HintedHandOffManager.access$200(HintedHandOffManager.java:84)
        at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:385)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
{noformat}

I added another FD check for this in the patch on CASSANDRA-3677 (which isn't the problem
there) but something similar should work.
                
> Multiple threads can attempt hint handoff to the same target
> ------------------------------------------------------------
>
>                 Key: CASSANDRA-3681
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3681
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>              Labels: hintedhandoff
>             Fix For: 1.0.7
>
>         Attachments: 3681.txt
>
>
> HintedHandOffManager attempts to prevent multiple threads sending hints to the same target
with the queuedDeliveries set, but the code is buggy.  If two handoffs *do* occur concurrently,
the second thread can use an arbitrarily large amount of memory skipping tombstones when it
starts paging from the beginning of the hint row, looking for the first live hint.  (This
is not a problem with a single thread, since it always pages starting with the last-seen hint
column name, effectively skipping the tombstones.  Then it compacts when it's done.)
> Technically this bug is present in all older Cassandra releases, but it only causes problems
in 1.0.x since the hint rows tend to be much larger (since there is one hint per write containing
the entire mutation, instead of just one per row consisting of just the key).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message