accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2288) Tserver under heavy ingest dies when unloading many tablets
Date Thu, 30 Jan 2014 19:10:13 GMT


Keith Turner commented on ACCUMULO-2288:

w/ 200MB mutations you could have a a lot of them in memory at any given time.  For example
each clients mutations will have  to be read into tserver memory, even if Accumulo is not
allowing any more writes because memory is full.  So w/ 10 clients it seems you could have
at least 10 200M mutations (in java memory) waiting to be added to native memory.  Then there
are the ones that have been added to the tablets memory (which is native memory).  Is this
a mutation with a 200M key/value?  If so minor compactions will have to read 200M from native
memory into java memory to write it out.  Also any major compactions or scans would be reading
these 200M values into java memory (probably at least 1 200M key/value per tablet file). 
Also these huge values may end up in the rfile index, using more memory when thats read into
java memory for scans or compactions.   

> Tserver under heavy ingest dies when unloading many tablets
> -----------------------------------------------------------
>                 Key: ACCUMULO-2288
>                 URL:
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.4.4
>         Environment: 4G heap, 6G native map for tserver;
> table.compaction.minor.logs.threshold=6
>            Reporter: Mike Drob
>         Attachments: logs.tar.gz
> # Create a table
> # Add lots of splits (they'll all be on the same server)
> # Start a heavy ingest load
> # Wait for master to decide to balance
> # When master sends unload requests, tablet server dies.
> It would be good if instead of trying to flush everything at once, the tserver could
go a little bit slower, but stay up. Or at least provide better logging.

This message was sent by Atlassian JIRA

View raw message