accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Fuchs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1052) Minor compactions not finishing before master kills tabletserver can very large number of files per tablet
Date Mon, 18 Feb 2013 18:57:13 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580772#comment-13580772
] 

Adam Fuchs commented on ACCUMULO-1052:
--------------------------------------

There are really only two reasons we have merging minor compactions -- the first is that HDFS
craps out when there are too many open files (like on a read of a tablet with lots and lots
of files). The second is that reading from a tablet with too many files can be slow. Merging
minor compactions have two effects. The first is to cap the number of files to prevent the
aforementioned crapping out. The second is to throttle long-term ingest so that it's tied
to disk I/O capacity. The problem with using merging minor compactions as a throttling mechanism
is that it actually generates more disk I/O load by making extra copies of entries (doing
N^2 work instead of NlogN in the worst case). We could probably use a better mechanism for
throttling. Meanwhile, an alternative solution might be to add more major compaction threads
and reduce the number of minor compaction threads.

When major compactions are keeping up, the smallest file should be really small. With a 100GB
tablet and a major compaction ratio of 3.0, the 15th largest file should be no more than 257MB
and the 20th largest file should be no bigger than about 34MB. With a max files limit of 20,
minor compactions should take on the order of 5 seconds in that scenario. This might be a
reason to bump up the max files threshold from the default of 15. Here's a chart that shows
a few file size stats, with major compaction ratio 3, assuming major compactions are keeping
up:
||#Files||Min(Max File Size/Min File Size)||Min(Tabet Size/Min File Size)||
|1|1|1|
|2|1|2|
|3|1|3|
|4|1.5|4.5|
|5|2.25|6.75|
|6|3.375|10.125|
|7|5.0625|15.1875|
|8|7.59375|22.78125|
|9|11.390625|34.171875|
|10|17.0859375|51.2578125|
|11|25.62890625|76.88671875|
|12|38.44335938|115.3300781|
|13|57.66503906|172.9951172|
|14|86.49755859|259.4926758|
|15|129.7463379|389.2390137|
|16|194.6195068|583.8585205|
|17|291.9292603|875.7877808|
|18|437.8938904|1313.681671|
|19|656.8408356|1970.522507|
|20|985.2612534|2955.78376|
|21|1477.89188|4433.67564|
|22|2216.83782|6650.51346|
|23|3325.25673|9975.77019|
|24|4987.885095|14963.65529|
|25|7481.827643|22445.48293|
|26|11222.74146|33668.22439|
|27|16834.1122|50502.33659|
|28|25251.16829|75753.50488|
|29|37876.75244|113630.2573|
|30|56815.12866|170445.386|
                
> Minor compactions not finishing before master kills tabletserver can very large number
of files per tablet
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: ACCUMULO-1052
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1052
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: master, tserver
>    Affects Versions: 1.4.2
>         Environment: Large, write-heavy cluster
>            Reporter: Josh Elser
>            Assignee: Eric Newton
>
> On a cluster that is being saturated with heavy ingest, a tserver is observed attempting
to perform a minor compaction for a tablet with multiple WALs. Because of this, commits to
this tablet end up being held.
> After churning on the minc for some time, the master's hold-time limit for tservers is
exceeded, however the minc didn't finish. The tserver is forcibly killed, the tablet is migrated,
recovery occurs on the new tserver and the problem repeats.
> Some of the minor compactions must finish, as the number of files for that tablet continue
to grow, but major compactions must not have time to finish since the number of files grow
unbounded.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message