accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2889) Batch metadata table updates for new walogs
Date Mon, 30 Jun 2014 16:46:26 GMT


Keith Turner commented on ACCUMULO-2889:

The calls to {{ThriftClientHandler.applyUpdates(TInfo, long, TKeyExtent, List<TMutation>)}}
are async thrift calls.  The client makes the calls against a sessionid.  On the tserver side
it buffers data for that sessionid (periodically dumping to walog and in-mem map depending
on memory use)

For Accismus I wrote a little wrapper that allows mulitple threads to submit mutations to
a batch writer and flush.   Something like this could possibly be used for what [~elserj]j
is suggesting for allowing multiple threads to batch writes to metadata.   This wrapper is

For my purposes this is still not where I want it to be, because a threads has to wait until
all other threads mutations are flushed.  Ideally once its mutations are flushed, it can continue.
 For batching writes to the metadata table, this case would matter when the metadata table
is split into many tablets. 

> Batch metadata table updates for new walogs
> -------------------------------------------
>                 Key: ACCUMULO-2889
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>    Affects Versions: 1.5.1, 1.6.0
>            Reporter: Jonathan Park
>            Assignee: Jonathan Park
>         Attachments: ACCUMULO-2889.0.patch.txt, ACCUMULO-2889.1.patch, accumulo-2889-withpatch.png,
> Currently, when we update the Metadata table with new loggers, we will update the metadata
for each tablet serially. We could optimize this to instead use a batchwriter to send all
metadata updates for all tablets in a batch.
> A few special cases include:
> - What if the !METADATA tablet was included in the batch?
> - What about the root tablet?
> Benefit:
> In one of our clusters, we're experiencing particularly slow HDFS operations leading
to large oscillations in ingest performance. We haven't isolated the cause in HDFS but when
we profile the tservers, we noticed that they were waiting for metadata table operations to
complete. This would target the waiting.
> Potential downsides:
> Given the existing locking scheme, it looks like we may have to lock a tablet for slightly
longer (we'll lock for the duration of the batch).

This message was sent by Atlassian JIRA

View raw message