accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3327) tablet server re-reads the bulk loaded flags with every bulk import request
Date Wed, 06 May 2015 18:53:00 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531184#comment-14531184
] 

Keith Turner commented on ACCUMULO-3327:
----------------------------------------

Was discussing w/ [~ecn] because I was concerned about the 4hr timeout.   We decided that
another solution is to drop the timeout and just periodically look in ZK to see what bulk
txs are active, removing flags for non-active txs from cache.   

A case where the 4hr timeout could cause problems is where the master is stopped for a long
period.  Since bulk import is a fate op, when the master starts it could resume a bulk import
that had been idle for a long period.

> tablet server re-reads the bulk loaded flags with every bulk import request
> ---------------------------------------------------------------------------
>
>                 Key: ACCUMULO-3327
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3327
>             Project: Accumulo
>          Issue Type: Sub-task
>          Components: tserver
>    Affects Versions: 1.5.1, 1.6.0, 1.6.1
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>             Fix For: 1.8.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> On a very large cluster, which bulk loads many thousands of files every few minutes,
I noticed the servers would reload the bulk imported flags with every request.  This put a
lot of pressure on the accumulo.metadata table, and it just isn't necessary: the tablet should
be tracking which bulk import files it has loaded, except when it first loaded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message