accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Fuchs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1755) BatchWriter blocks all addMutation calls while binning mutations
Date Wed, 24 Feb 2016 19:25:18 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15163556#comment-15163556
] 

Adam Fuchs commented on ACCUMULO-1755:
--------------------------------------

Dave, thanks for resurrecting this ticket! Looks like your implementation is oriented towards
the "don't hold a lock while binning" branch. This should work well for a multi-threaded client
sharing a BatchWriter, but probably won't affect the performance of a single-threaded client.
Do you have any perf tests you're using to evaluate how well your solution works? I don't
remember whether any of our examples use a multi-threaded "mutation producer" pattern.

One thing we might consider is whether we want to try to optimize the case of users writing
simpler single-threaded clients and letting Accumulo handle all of the parallelization. That
might lead us to an alternative implementation of moving the binning operation to another
thread.

> BatchWriter blocks all addMutation calls while binning mutations
> ----------------------------------------------------------------
>
>                 Key: ACCUMULO-1755
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1755
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client
>            Reporter: Adam Fuchs
>            Assignee: Dave Marion
>
> Through code inspection, we found that the BatchWriter bins mutations inside of a synchronized
block that covers calls to addMutation. Binning potentially involves lookups of tablet metadata
and processes a fair amount of information. We will get better parallelism if we can either
unlock the lock while binning, dedicate another thread to do the binning, or use one of the
send threads to do the binning.
> This has not been verified empirically yet, so there is not yet any profiling info to
indicate the level of improvement that we should expect. Profiling and repeatable demonstration
of this performance bottleneck should be the first step on this ticket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message