accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Fuchs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1085) make the number of threads for assignment configurable
Date Tue, 12 Mar 2013 01:15:12 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599547#comment-13599547
] 

Adam Fuchs commented on ACCUMULO-1085:
--------------------------------------

We ran an unintentional test the other day where we ran tdown on an actively ingesting cluster
with lots of memory, and recovery of the tablets was too slow to wait for (would have taken
many hours). I think that's a really good reason to try to get this into 1.5. Has anyone else
experienced similar problems?

bq. When walogs are recovered, they are immediately compacted. This is done to keep from running
out of memory, in the case where multiple large in memory maps need to be recovered. Having
multiple threads doing log recovery could possibly exhaust memory. The code that does recovery
would need to defend against this in some way.

Can we flush tablets that already have a lot buffered (i.e. those that are not being recovered)?
Writing really small RFiles might be a source of inefficiency that is leading to slow recovery
times, although I have no evidence to back that up.
                
> make the number of threads for assignment configurable
> ------------------------------------------------------
>
>                 Key: ACCUMULO-1085
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1085
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: tserver
>            Reporter: Eric Newton
>            Assignee: Keith Turner
>             Fix For: 1.6.0
>
>         Attachments: ACCUMULO-1085.patch
>
>
> Log recoveries from sorted logs take considerable time: run them in parallel with more
assignment threads.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message