accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-4187) Rate limiting of major compactions
Date Tue, 12 Apr 2016 20:02:25 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237869#comment-15237869
] 

ASF GitHub Bot commented on ACCUMULO-4187:
------------------------------------------

GitHub user ShawnWalker opened a pull request:

    https://github.com/apache/accumulo/pull/90

    ACCUMULO-4187: Added rate limiting for major compactions.

    Added configuration property `tserver.compaction.major.throughput` of type `PropertyType.MEMORY`
to control rate limiting of major compactions on each tserver.
      
    Specifying a value of `0B` (the default) disables rate limiting.
    
    If a positive value is specified, then all tablet servers will limit the I/O performed
during major compaction accordingly.  For example, with `tserver.compaction.major.throughput=30M`,
then each tserver will read no more than 30MiB per second and write no more than 30MiB combined
over all major compaction threads.
    
    This change involved adding an optional `RateLimiter` parameter to `FileOperations.openReader(...)`
and `FileOperations.openWriter(...)`.  Most of the file changes involve adding an appropriate
`null` to invocations of these methods.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ShawnWalker/accumulo ACCUMULO-4187

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/accumulo/pull/90.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #90
    
----
commit 51396335b275b980c09964ad3ca95bf2a6fe5842
Author: Shawn Walker <accumulo@shawn-walker.net>
Date:   2016-04-06T17:18:09Z

    ACCUMULO-4187: Added rate limiting for major compactions.
    
    Added configuration property tserver.compaction.major.throughput of type PropertyType.MEMORY
with a default of 0B (unlimited).  If another value is specified (e.g. 30M), then all tablet
servers will limit the I/O performed during major compaction accordingly (e.g. neither reading
nor writing more than 30MiB per second combined over all major compaction threads).

----


> Rate limiting of major compactions
> ----------------------------------
>
>                 Key: ACCUMULO-4187
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4187
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: core, tserver
>    Affects Versions: 1.8.0
>            Reporter: Shawn Walker
>            Priority: Minor
>
> In discussing [ACCUMULO-4166] with Keith Turner, we decided that the underlying issue
is that major compactions can overwhelm a tablet server, rendering it nearly unresponsive.
> To address this, we should take a cue from Apache Cassandra and restrict how quickly
we perform major compactions.  Rate limiting reads and writes involved in major compactions
will directly affect the IO load caused by major compactions, and should also indirectly affect
the CPU load.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message