cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Hobbs (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-11320) Improve backoff policy for cqlsh COPY FROM
Date Wed, 30 Mar 2016 18:55:25 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tyler Hobbs updated CASSANDRA-11320:
------------------------------------
    Status: Open  (was: Patch Available)

> Improve backoff policy for cqlsh COPY FROM
> ------------------------------------------
>
>                 Key: CASSANDRA-11320
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11320
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Stefania
>            Assignee: Stefania
>              Labels: doc-impacting
>             Fix For: 3.0.x, 3.x
>
>
> Currently we have an exponential back-off policy in COPY FROM that kicks in when timeouts
are received. However there are two limitations:
> * it does not cover new requests and therefore we may not back-off sufficiently to give
time to an overloaded server to recover
> * the pause is performed in the receiving thread and therefore we may not process server
messages quickly enough
> There is a static throttling mechanism in rows per second from feeder to worker processes
(the INGESTRATE) but the feeder has no idea of the load of each worker process. However it's
easy to keep track of how many chunks a worker process has yet to read by introducing a bounded
semaphore.
> The idea is to move the back-off pauses to the worker processes main thread so as to
include all messages, new and retries, not just the retries that timed out. The worker process
will not read new chunks during the back-off pauses, and the feeder process can then look
at the number of pending chunks before sending new chunks to a worker process.
> [~aholmber], [~aweisberg] what do you think?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message