accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Vines (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-902) Have a common resource pool for minor and major compactions
Date Wed, 04 Sep 2013 22:42:54 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758464#comment-13758464
] 

John Vines commented on ACCUMULO-902:
-------------------------------------

So, I put some thought into this and I think the easiest way to configure it is to define
a minimum and maximum amount of MinC and MajC. Then the overall maximum amount of *Compactions
is the max(minimum MinC + maximum MajC, minimum MajC + maximum MinC) [Though I would think
that the latter argument is dominant due to suggested configurations]. Then the weights of
minor vs. major can be determined by the comparing the difference of max and min of each type
of compaction.

The tricky part, aside from setting up the thread pool(s) to use and managing it, is determining
which task to go for next. Of course, if there are < minimum of a type running, those should
jump to the front of the queue. But depending on your configuration, there is the possibility
that Majors get blocked off because minors keep taking up slots (they wrap and then no major
can fit in that slot so another minor runs, etc. so Majors get bound by the minimum, though
this could be the case of a bad configuration).

An optional component is to allow some customization in the order of compactions to allow
some sort of per-table QoS (which may be a refactoring of the !METADATA/!!ROOT table code),
but that's another conversation.

There also lends the case of a cost of merging minor compactions vs. normal minor compactions,
but I think its' really not worth segregating them.
                
> Have a common resource pool for minor and major compactions
> -----------------------------------------------------------
>
>                 Key: ACCUMULO-902
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-902
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: John Vines
>            Assignee: Keith Turner
>             Fix For: 1.6.0
>
>
> Currently we have a defined threadpool for minor and major compactions, independent of
one another. However, there are situations where a system may be minor compaction heavy with
no major, or vice versa. I would like to see a common threadpool which is accessible to both
operations for work to be done, with guarantees for certain resources to be available to the
other type of work. That is, it should be a defined pool size with a (configurable) minimum
of resources maintained for the other to maintain a certain QoS. Of course, major is heavier
than minor, so some weighting of operations needs to be done to keep workloads reasonable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message