accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2889) Batch metadata table updates for new walogs
Date Mon, 30 Jun 2014 16:28:25 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047813#comment-14047813
] 

Josh Elser commented on ACCUMULO-2889:
--------------------------------------

I've been looking over the patch after seeing the lower performance numbers than I would have
hoped to see.

I'm wary of using the ThreadLocal as a way to synchronize updates. I don't think we're getting
as much batching as we could because the tabletserver is going to be running multiple clients
in their own threads. You'll only get batching when a single write session writes to multiple
tablets on that one tserver. I think trying to synchronize access around the InternalBatchWriter
class (or use a concurrent data structure as the queue) might be cleaner to understand and
would cache across different threads. Switching to a concurrent data structure would also
need some extra synchronization around {{commitBatch()}} as you'd have a race condition on
clearing the data structure after a {{flush()}} on the underlying batchwriter.

Actually, looking at {{ThriftClientHandler.applyUpdates(TInfo, long, TKeyExtent, List<TMutation>)}},
I'm not sure you're going to be getting any batching at all. The tserver is only receiving
updates for a single tablet in one thrift call (a thread) which means that all writes to multiple
tablets are in different threads. I could be missing something, but that might drive home
the point.

Ideally, you'd want something that can very quickly (maybe even lock free somehow?) add Mutations
that need to be {{flush()}}'ed and then get a single notification point that all of the threads
could wait on to know that the sync happened (a CountdownLatch, perhaps).

> Batch metadata table updates for new walogs
> -------------------------------------------
>
>                 Key: ACCUMULO-2889
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2889
>             Project: Accumulo
>          Issue Type: Improvement
>    Affects Versions: 1.5.1, 1.6.0
>            Reporter: Jonathan Park
>            Assignee: Jonathan Park
>         Attachments: ACCUMULO-2889.0.patch.txt, ACCUMULO-2889.1.patch, accumulo-2889-withpatch.png,
accumulo-2889_withoutpatch.png, batch_perf_test.sh, run_all.sh, start-ingest.sh
>
>
> Currently, when we update the Metadata table with new loggers, we will update the metadata
for each tablet serially. We could optimize this to instead use a batchwriter to send all
metadata updates for all tablets in a batch.
> A few special cases include:
> - What if the !METADATA tablet was included in the batch?
> - What about the root tablet?
> Benefit:
> In one of our clusters, we're experiencing particularly slow HDFS operations leading
to large oscillations in ingest performance. We haven't isolated the cause in HDFS but when
we profile the tservers, we noticed that they were waiting for metadata table operations to
complete. This would target the waiting.
> Potential downsides:
> Given the existing locking scheme, it looks like we may have to lock a tablet for slightly
longer (we'll lock for the duration of the batch).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message