accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Fuchs (JIRA)" <>
Subject [jira] [Updated] (ACCUMULO-1755) BatchWriter blocks all addMutation calls while binning mutations
Date Tue, 08 Mar 2016 14:54:40 GMT


Adam Fuchs updated ACCUMULO-1755:
                Screen Shot 2016-03-08 at 9.47.39 AM.png

[~dlmarion] et. al., I did a bit of theory work around this and built a simulator to illustrate
performance with different pipelining strategies. Here are my calculations to go with it:

N = Batch Size
S = Send Time per mutation
B = Binning Time per Mutation
G = Generation Time per Mutation
T = # Send Threads
P = # Generation Threads

Current 2-Stage Pipeline Throughput
= N / max(stage 1 time, stage 2 time)
= N / max(BN+GN/P, SN/T)
= 1 / max(B+G/P, S/T)
= min(1/(B+G/P), T/S)

Async 3-Stage Pipeline Throughput
= N / max(stage1, stage2, stage3)
= N / max(BN, GN/P, SN/T)
= 1 / max(B, G/P, S/T)
= min(1/B, P/G, T/S)

(Assumes no cpu contention, even workload over threads)

Theoretical implications include:
* When binning time, per-thread generation time, and per-thread send time are close we can
speed up the pipeline by 2x by adding another stage.
* We can balance the pipeline between generation threads and send threads but not binning
threads – may need a pool for that to further optimize TSBW throughput
* If send threads or downstream elements are the bottleneck then we will have less impact
by adding a third stage to the pipeline

> BatchWriter blocks all addMutation calls while binning mutations
> ----------------------------------------------------------------
>                 Key: ACCUMULO-1755
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client
>            Reporter: Adam Fuchs
>            Assignee: Dave Marion
>             Fix For: 1.6.6, 1.7.2, 1.8.0
>         Attachments: 1755-nosync-perf-test.patch, 1755-perf-test.patch, ACCUMULO-1755.patch,
Screen Shot 2016-03-08 at 9.47.39 AM.png,
>          Time Spent: 2h
>  Remaining Estimate: 0h
> Through code inspection, we found that the BatchWriter bins mutations inside of a synchronized
block that covers calls to addMutation. Binning potentially involves lookups of tablet metadata
and processes a fair amount of information. We will get better parallelism if we can either
unlock the lock while binning, dedicate another thread to do the binning, or use one of the
send threads to do the binning.
> This has not been verified empirically yet, so there is not yet any profiling info to
indicate the level of improvement that we should expect. Profiling and repeatable demonstration
of this performance bottleneck should be the first step on this ticket.

This message was sent by Atlassian JIRA

View raw message