accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1052) Minor compactions not finishing before master kills tabletserver can very large number of files per tablet
Date Fri, 08 Feb 2013 20:45:12 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13574805#comment-13574805
] 

Josh Elser commented on ACCUMULO-1052:
--------------------------------------

Merging minor compactions were indeed disabled (max files increased drastically) due to the
extremely long minor compaction when ingesting into a table which already has decently sized
rfiles. Thinking about this makes me think we need a ticket to configure a maximum size of
files in which to actually perform merging minor compactions on. Meaning, when a small amount
of data is waiting to be minor compacted, and the smallest file which already exists for that
tablet is very large (smallest among many very large files), merging minor compactions really
suck.
                
> Minor compactions not finishing before master kills tabletserver can very large number
of files per tablet
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: ACCUMULO-1052
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1052
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: master, tserver
>    Affects Versions: 1.4.2
>         Environment: Large, write-heavy cluster
>            Reporter: Josh Elser
>            Assignee: Eric Newton
>
> On a cluster that is being saturated with heavy ingest, a tserver is observed attempting
to perform a minor compaction for a tablet with multiple WALs. Because of this, commits to
this tablet end up being held.
> After churning on the minc for some time, the master's hold-time limit for tservers is
exceeded, however the minc didn't finish. The tserver is forcibly killed, the tablet is migrated,
recovery occurs on the new tserver and the problem repeats.
> Some of the minor compactions must finish, as the number of files for that tablet continue
to grow, but major compactions must not have time to finish since the number of files grow
unbounded.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message