accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Marion (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1755) BatchWriter blocks all addMutation calls while binning mutations
Date Tue, 08 Mar 2016 14:50:40 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184994#comment-15184994
] 

Dave Marion commented on ACCUMULO-1755:
---------------------------------------

I wrote a new test that sends 1M mutations total using N threads with a BatchWriter buffer
of different sizes. The test is run twice and the time discarded to account for JVM startup.
Then the test is run 10 times and the average (in seconds) is reported for total time and
time to add mutations. 

First, I added some code to the TSBW to determine that with my test data I was sending the
following number of batches using 1, 10, and 100MB buffers:

||BatchWriter Max Memory Size || Flushes To Accumulo
| 1MB   |  515 |
| 10MB  |  52  |
| 100MB |  6   |

Here are the results of the test:

h2. master branch

Using the patch 1755-perf-test.patch

Total Time

||Threads|| 1MB || 10MB || 100MB ||
| 1 | 3.121  | 2.818 | 3.158 |
| 2 | 3.102 | 2.414 | 2.950 |
| 4 | 3.367 | 2.573 | 3.114 |
| 8 | 3.422 | 2.569 | 3.140 |
| 16 | 3.590 | 2.741 | 3.332 |

Add Mutation Time

||Threads|| 1MB || 10MB || 100MB ||
| 1 | 3.114 | 2.733 | 2.498 |
| 2 | 3.088 | 2.350 | 2.371 |
| 4 | 3.360 | 2.506 | 2.472 |
| 8 | 3.414 | 2.516 | 2.509 |
| 16 | 3.582 | 2.692 | 2.696 |

h2. master branch with modifications to remove sync on addMutation()

I successfully modified the TSBW to remove the synchronization modifier from the addMutation
method. The multi-threaded binning test passes so I have some confidence that the data is
correct. Use patch 1755-nosync-perf-test.patch

Total Time

||Threads|| 1MB || 10MB || 100MB ||
| 1 | 3.080 | 2.766 | 3.255 |
| 2 | 2.972 | 2.420 | 3.137 |
| 4 | 3.162 | 2.492 | 3.190 |
| 8 | 3.100 | 2.658 | 3.623 |
| 16 | 3.393 | 2.898 | 3.743 |

Add Mutation Time

||Threads|| 1MB || 10MB || 100MB ||
| 1 | 3.072 | 2.653 | 2.517 |
| 2 | 2.965 | 2.371 | 2.527 |
| 4 | 3.155 | 2.441 | 2.589 |
| 8 | 3.092 | 2.602 | 2.961 |
| 16 | 3.385 | 2.839 | 2.891 |

 I think the results are inconclusive. The tests run with MAC on localhost, so this is likely
a best case scenario. I'd be interested to see this re-run on a real cluster.


> BatchWriter blocks all addMutation calls while binning mutations
> ----------------------------------------------------------------
>
>                 Key: ACCUMULO-1755
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1755
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: client
>            Reporter: Adam Fuchs
>            Assignee: Dave Marion
>             Fix For: 1.6.6, 1.7.2, 1.8.0
>
>         Attachments: ACCUMULO-1755.patch
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> Through code inspection, we found that the BatchWriter bins mutations inside of a synchronized
block that covers calls to addMutation. Binning potentially involves lookups of tablet metadata
and processes a fair amount of information. We will get better parallelism if we can either
unlock the lock while binning, dedicate another thread to do the binning, or use one of the
send threads to do the binning.
> This has not been verified empirically yet, so there is not yet any profiling info to
indicate the level of improvement that we should expect. Profiling and repeatable demonstration
of this performance bottleneck should be the first step on this ticket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message