accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-4021) bulk imports slow file garbage collection
Date Tue, 20 Oct 2015 19:51:27 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965645#comment-14965645
] 

Eric Newton commented on ACCUMULO-4021:
---------------------------------------

It was pointed out to me: if a bulk file is consistently loaded into the tableId that sorts
last before a full metadata table scan can finish, the GC run will never complete.


> bulk imports slow file garbage collection
> -----------------------------------------
>
>                 Key: ACCUMULO-4021
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-4021
>             Project: Accumulo
>          Issue Type: Bug
>          Components: gc
>    Affects Versions: 1.6.3
>         Environment: large production system
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>             Fix For: 1.6.5, 1.7.1, 1.8.0
>
>
> On a large system, bulk imports slow file garbage collection to a crawl.  The total number
of files to be deleted was about 14 million. Initially, it would run quickly, but then slow
down, to the point where only a few files would be deleted every few minutes. The jvm was
only using 50% of the CPU (and therefore, probably not GC thrashing). JStacks showed the collector
scanning the metadata table to remove referenced files from the delete list.
> If the bulk ingest requests were stopped, the GC completed quickly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message