accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2368) Addsplits to an offline table
Date Thu, 26 Jun 2014 01:11:26 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044241#comment-14044241
] 

Keith Turner commented on ACCUMULO-2368:
----------------------------------------

I wrote a little [utility|https://gist.github.com/keith-turner/5c561e438cb04c501b6e] to time
splitting and balancing.  I ran this utility on EC2 using 1.6.1-SNAPSHOT (76c910b + the [utility|https://gist.github.com/keith-turner/5c561e438cb04c501b6e]).
 I used 20 m1.large instances w/ 17 tservers.   

{noformat}
$ ./bin/accumulo org.apache.accumulo.test.SplitTimeTest -u root -p secret --numTablets 10000
split time: 72.26 secs  balance time: 113.98 secs
total time: 186.24 secs
$ ./bin/accumulo org.apache.accumulo.test.SplitTimeTest -u root -p secret --numTablets 50000
split time: 518.98 secs  balance time: 428.80 secs
total time: 947.78 secs
{noformat} 

The utility has an option to presplit and wait for the initial tablets to balance before doing
all splits. This option made a noticeable difference in the total time.

{noformat}
$ ./bin/accumulo org.apache.accumulo.test.SplitTimeTest -u root -p secret --preSplit 34 --numTablets
10000
pre split time: 0.74 secs  balance time: 14.86 secs
split time: 74.02 secs  balance time: 14.10 secs
total time: 103.72 secs
{noformat}

I suspected that hsync was slowing splitting down (slowing down metadata write made during
splitting), so I tried switching to hflush and saw no difference on EC2.   I did some targeted
experiments and found that hflush and hsync performance were roughly the same on EC2.   I
was using ephemeral storage for hdfs. 

On my workstation the performance of hflush and hsync is very different.  The following experiment
was run on my workstation w/ a single tablet server w/ hsync. 

{noformat}
$ ./bin/accumulo org.apache.accumulo.test.SplitTimeTest -u root -p secret --numTablets 1000
split time: 38.05 secs  balance time: 0.07 secs
total time: 38.13 secs
{noformat}

Below is after setting {{config -s tserver.wal.sync.method=hflush}} in the shell and restarting.
 Made a big difference.

{noformat}
$ ./bin/accumulo org.apache.accumulo.test.SplitTimeTest -u root -p secret --numTablets 1000
split time: 3.86 secs  balance time: 0.06 secs
total time: 3.92 secs
{noformat}

Curious if 1.6.0 is even slower splitting w/ hsync because of ACCUMULO-2766.

Offline split would probably be faster, it would not need to contend with as many hsyncs.
  For offline splits to be useful,  it seems like offlineTable+offlineSplit+onlineTable would
need to be faster than onlineSplit.   offline and online table may still run into issues w/
lots of hsyncs caused by metadata updates made when tablets load and unload.  Would be good
to look into the times of offline and online table.

An option for speeding up online splits may be to add more threads to the code that does splits.
  This along w/ group commit on the metadata table walog may allow more split related updates
to be made per hsync.   I am going to experiment w/ this.

> Addsplits to an offline table
> -----------------------------
>
>                 Key: ACCUMULO-2368
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2368
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: master
>            Reporter: John Vines
>            Assignee: Sean Busbey
>             Fix For: 1.7.0
>
>
> Currently a table must be online to addsplits. Firstly, it's relatively slow. Secondly,
it could be a LOT faster to do it to an offline table because it's just a few metadata writes
per split point.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message