accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3327) tablet server re-reads the bulk loaded flags with every bulk import request
Date Fri, 08 May 2015 22:43:00 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535792#comment-14535792
] 

Keith Turner commented on ACCUMULO-3327:
----------------------------------------

I was thinking about this approach of periodically looking in zookeeper.  I realized that
there could be the following race condition.

 # THREAD1 : look in zookeeper to get active set of transactions, getting TX1 and TX2
 # THREAD2 : bulk import TX3 starts and its id is added to cache
 # THREAD1 : removes TX3 from cache because "its not in ZK"

I think an easy to work around is to get the set of unique IDs in the cache before looking
in ZK.  Then only remove txids from the cache that were in the cache before looking in ZK.

> tablet server re-reads the bulk loaded flags with every bulk import request
> ---------------------------------------------------------------------------
>
>                 Key: ACCUMULO-3327
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3327
>             Project: Accumulo
>          Issue Type: Sub-task
>          Components: tserver
>    Affects Versions: 1.5.1, 1.6.0, 1.6.1
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>             Fix For: 1.8.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> On a very large cluster, which bulk loads many thousands of files every few minutes,
I noticed the servers would reload the bulk imported flags with every request.  This put a
lot of pressure on the accumulo.metadata table, and it just isn't necessary: the tablet should
be tracking which bulk import files it has loaded, except when it first loaded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message