accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Marion (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-1755) BatchWriter blocks all addMutation calls while binning mutations
Date Wed, 02 Mar 2016 23:34:18 GMT


Dave Marion commented on ACCUMULO-1755:

I took the test that I created and ran it against master and my feature branch with 1 to 6
threads. I didn't see much difference, but looking back at it now I think its because the
test pre-creates all of the mutations and adds them as fast as possible. The test is really
for multi-threaded correctness rather than performance. In the new code there is still a synchronization
point when adding the binned mutations to the queues for the tablet servers. The send threads
in the test (local mini accumulo cluster) must be able to keep up with adding of the binned
mutations. I don't expect that to be the case in a real deployment. Good news - performance
wasn't worse.

I think a better test is to write a simple multi-threaded client that creates and adds mutations
to a common batch writer. Then, time the application as whole trying to insert N mutations
with 1 to N client threads. The previous implementation blocked all client threads from calling
BatchWriter.addMutation(), meaning the clients could not do any work. In the new implementation
the clients will be able to continue to do work, adding mutations, and even binning them in
their own thread if necessary, before blocking. I'll see if I can re-test with this new approach
in the next few days. Do you have a different thought about how to test this?

> BatchWriter blocks all addMutation calls while binning mutations
> ----------------------------------------------------------------
>                 Key: ACCUMULO-1755
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client
>            Reporter: Adam Fuchs
>            Assignee: Dave Marion
>             Fix For: 1.6.6, 1.7.2, 1.8.0
>         Attachments: ACCUMULO-1755.patch
>          Time Spent: 2h
>  Remaining Estimate: 0h
> Through code inspection, we found that the BatchWriter bins mutations inside of a synchronized
block that covers calls to addMutation. Binning potentially involves lookups of tablet metadata
and processes a fair amount of information. We will get better parallelism if we can either
unlock the lock while binning, dedicate another thread to do the binning, or use one of the
send threads to do the binning.
> This has not been verified empirically yet, so there is not yet any profiling info to
indicate the level of improvement that we should expect. Profiling and repeatable demonstration
of this performance bottleneck should be the first step on this ticket.

This message was sent by Atlassian JIRA

View raw message