cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-13055) DoS by StreamReceiveTask, during incremental repair
Date Tue, 20 Dec 2016 20:49:58 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765213#comment-15765213
] 

Paulo Motta edited comment on CASSANDRA-13055 at 12/20/16 8:49 PM:
-------------------------------------------------------------------

It seems this was mostly caused by multiple streaming sessions being executed simultaneously
for the same repair job due to {{RepairJob}} releasing the executor after [validations were
finished|https://github.com/apache/cassandra/blob/732af7de7f4b7865c00dfa0d85e4dbf4ee9900e2/src/java/org/apache/cassandra/repair/RepairJob.java#L160],
which means the next repair session starts while the previous is still syncing. This should
be alleviated significantly on 3.0.11 by CASSANDRA-12901, which waits for a previous {{RepairSession}}
to finish (sync streaming included) before starting the repair session for the next vnode.
Can you test with that version to see if it improves things?

Another thing is that CASSANDRA-7795 added {{StreamReceiveTask}} executor's limited by {{FBUtilities.getAvailableProcessors()}},
but this limit was removed by CASSANDRA-6455 without clear reason, so I believe it could be
a merge error. Do you confirm [~yukim] or is there any particular reason for removing this
executor's limit?

If the limit removal was unintentional, I propose restoring it to {{FBUtilities.getAvailableProcessors()}}
and reduce it's priority, while possibly making it settable via JMX or system property for
advanced users which want more control, but I think it shouldn't be necessary in most cases
specially after the additional repair job synchronization added by CASSANDRA-12901.
* Please note that this is not a limit for concurrent streaming tasks, but only to the final
completion phase of streaming, where sstables are added to the tracker which can be CPU intensive,
but should not dominate streaming time in general.
** Nevertherless it would be nice to check the impact of this for bootstrap, I believe it
won't have any significant impact but it's good to test/verify.


was (Author: pauloricardomg):
It seems this was mostly caused by multiple streaming sessions being executed simultaneously
for the same repair job due to {{RepairJob}} releasing the executor after [validations were
finished|https://github.com/apache/cassandra/blob/732af7de7f4b7865c00dfa0d85e4dbf4ee9900e2/src/java/org/apache/cassandra/repair/RepairJob.java#L160],
which means the next repair session starts while the previous is still syncing. This should
be alleviated significantly on 3.0.11 by CASSANDRA-12901, which waits for a previous {{RepairSession}}
to finish before starting the repair session for the next vnode. Can you test with that version
to see if it improves things?

Another thing is that CASSANDRA-7795 added {{StreamReceiveTask}} executor's limited by {{FBUtilities.getAvailableProcessors()}},
but this limit was removed by CASSANDRA-6455 without clear reason, so I believe it could be
a merge error. Do you confirm [~yukim] or is there any particular reason for removing this
executor's limit?

If the limit removal was unintentional, I propose restoring it to {{FBUtilities.getAvailableProcessors()}}
and reduce it's priority, while possibly making it settable via JMX or system property for
advanced users which want more control, but I think it shouldn't be necessary in most cases
specially after the additional repair job synchronization added by CASSANDRA-12901.
* Please note that this is not a limit for concurrent streaming tasks, but only to the final
completion phase of streaming, where sstables are added to the tracker which can be CPU intensive,
but should not dominate streaming time in general.
** Nevertherless it would be nice to check the impact of this for bootstrap, I believe it
won't have any significant impact but it's good to test/verify.

> DoS by StreamReceiveTask, during incremental repair
> ---------------------------------------------------
>
>                 Key: CASSANDRA-13055
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13055
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Tom van der Woerdt
>         Attachments: untitled 2.txt
>
>
> There's no limit on how many StreamReceiveTask there can be, and during an incremental
repair on a vnode cluster with high replication factors, this can lead to thousands of conccurent
StreamReceiveTask threads, effectively DoSing the node.
> I just found one of my nodes with 1000+ loadavg, caused by 1363 concurrent StreamReceiveTask
threads.
> That sucks :)
> I think :
> * Cassandra shouldn't allow more than X concurrent StreamReceiveTask threads
> * StreamReceiveTask threads should be at a lower priority, like compaction threads
> Alternative ideas welcome as well, of course.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message