cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vincent White (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13797) RepairJob blocks on syncTasks
Date Thu, 01 Mar 2018 23:18:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382839#comment-16382839
] 

Vincent White commented on CASSANDRA-13797:
-------------------------------------------

Now that we don't wait for the validations of each repair job to finish before moving onto
the next one, I don't see anything to stop the repair coordinator from spinning through all
the token ranges and effectively triggering all the validations tasks at once, which could
be a significant amount of validation compactions on each node depending on your topology
and common ranges for that keyspace. I'm also not sure of the overhead of creating all the
futures/listeners on the coordinator at once in this case. 

In 3 the validation executor thread pool has no size limit so a new validation is started
as soon as a validation request is received. I admit I haven't caught up on the changes to
repair in trunk, and while the validation executor pool size is configurable in trunk, its
default is still Integer.MAX_VALUE.

I understand this same affect (hundreds of concurrent validations) can still happen if you
trigger a repair across a keyspace with a large number of column families but with this change
there is no way of avoiding it without using subrange repairs on a single column family (if
you have a topology/replication that cant be merged into a small number of common ranges)
.


> RepairJob blocks on syncTasks
> -----------------------------
>
>                 Key: CASSANDRA-13797
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13797
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Repair
>            Reporter: Blake Eggleston
>            Assignee: Blake Eggleston
>            Priority: Major
>             Fix For: 3.0.15, 3.11.1, 4.0
>
>
> The thread running {{RepairJob}} blocks while it waits for the validations it starts
to complete ([see here|https://github.com/bdeggleston/cassandra/blob/9fdec0a82851f5c35cd21d02e8c4da8fc685edb2/src/java/org/apache/cassandra/repair/RepairJob.java#L185]).
However, the downstream callbacks (ie: the post-repair cleanup stuff) aren't waiting for {{RepairJob#run}}
to return, they're waiting for a result to be set on RepairJob the future, which happens after
the sync tasks have completed. This post repair cleanup stuff also immediately shuts down
the executor {{RepairJob#run}} is running in. So in noop repair sessions, where there's nothing
to stream, I'm seeing the callbacks sometimes fire before {{RepairJob#run}} wakes up, and
causing an {{InterruptedException}} is thrown.
> I'm pretty sure this can just be removed, but I'd like a second opinion. This appears
to just be a holdover from before repair coordination became async. I thought it might be
doing some throttling by blocking, but each repair session gets it's own executor, and validation
is  throttled by the fixed size executors doing the actual work of validation, so I don't
think we need to keep this around.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message