accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2842) Allow for use of hflush instead of hsync
Date Fri, 23 May 2014 20:13:02 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007642#comment-14007642
] 

Josh Elser commented on ACCUMULO-2842:
--------------------------------------

Aforementioned "contrived benchmarks"

{noformat}
    1.5.2-SNAPSHOT (74acc30), 5 process, each writing 1M entries, with 500B values. TServer
running with 256M max WAL size, 4M tserver.mutation.queue.max, on Apache Hadoop 2.4.0
     
    With HSync
    -------------
    Run 1:
    1,000,000 records written | 9,543 records/sec | 529,000,000 bytes written | 5,048,673
bytes/sec | 104.780 secs
    1,000,000 records written | 9,523 records/sec | 529,000,000 bytes written | 5,038,095
bytes/sec | 105.000 secs
    1,000,000 records written | 9,377 records/sec | 529,000,000 bytes written | 4,960,754
bytes/sec | 106.637 secs
    1,000,000 records written | 9,355 records/sec | 529,000,000 bytes written | 4,949,244
bytes/sec | 106.885 secs
    1,000,000 records written | 9,037 records/sec | 529,000,000 bytes written | 4,781,056
bytes/sec | 110.645 secs
     
    Run 2:
    1,000,000 records written | 8,981 records/sec | 529,000,000 bytes written | 4,750,956
bytes/sec | 111.346 secs
    1,000,000 records written | 8,870 records/sec | 529,000,000 bytes written | 4,692,461
bytes/sec | 112.734 secs
    1,000,000 records written | 8,873 records/sec | 529,000,000 bytes written | 4,694,210
bytes/sec | 112.692 secs
    1,000,000 records written | 8,807 records/sec | 529,000,000 bytes written | 4,659,109
bytes/sec | 113.541 secs
    1,000,000 records written | 8,666 records/sec | 529,000,000 bytes written | 4,584,373
bytes/sec | 115.392 secs
     
    Run 3:
    1,000,000 records written | 8,524 records/sec | 529,000,000 bytes written | 4,509,611
bytes/sec | 117.305 secs
    1,000,000 records written | 8,389 records/sec | 529,000,000 bytes written | 4,438,031
bytes/sec | 119.197 secs
    1,000,000 records written | 8,337 records/sec | 529,000,000 bytes written | 4,410,317
bytes/sec | 119.946 secs
    1,000,000 records written | 8,312 records/sec | 529,000,000 bytes written | 4,397,559
bytes/sec | 120.294 secs
    1,000,000 records written | 8,276 records/sec | 529,000,000 bytes written | 4,378,088
bytes/sec | 120.829 secs
     
    Run 4:
    1,000,000 records written | 9,318 records/sec | 529,000,000 bytes written | 4,929,321
bytes/sec | 107.317 secs
    1,000,000 records written | 8,872 records/sec | 529,000,000 bytes written | 4,693,544
bytes/sec | 112.708 secs
    1,000,000 records written | 8,860 records/sec | 529,000,000 bytes written | 4,687,430
bytes/sec | 112.855 secs
    1,000,000 records written | 8,666 records/sec | 529,000,000 bytes written | 4,584,810
bytes/sec | 115.381 secs
    1,000,000 records written | 8,247 records/sec | 529,000,000 bytes written | 4,363,138
bytes/sec | 121.243 secs
     
    With HFlush:
    ===============
    Run 1:
    1,000,000 records written | 12,828 records/sec | 529,000,000 bytes written | 6,786,140
bytes/sec | 77.953 secs
    1,000,000 records written | 12,801 records/sec | 529,000,000 bytes written | 6,771,893
bytes/sec | 78.117 secs
    1,000,000 records written | 12,759 records/sec | 529,000,000 bytes written | 6,749,687
bytes/sec | 78.374 secs
    1,000,000 records written | 12,730 records/sec | 529,000,000 bytes written | 6,734,392
bytes/sec | 78.552 secs
    1,000,000 records written | 12,725 records/sec | 529,000,000 bytes written | 6,731,564
bytes/sec | 78.585 secs
     
    Run 2:
    1,000,000 records written | 12,046 records/sec | 529,000,000 bytes written | 6,372,572
bytes/sec | 83.012 secs
    1,000,000 records written | 12,016 records/sec | 529,000,000 bytes written | 6,356,492
bytes/sec | 83.222 secs
    1,000,000 records written | 12,012 records/sec | 529,000,000 bytes written | 6,354,507
bytes/sec | 83.248 secs
    1,000,000 records written | 12,002 records/sec | 529,000,000 bytes written | 6,349,092
bytes/sec | 83.319 secs
    1,000,000 records written | 11,985 records/sec | 529,000,000 bytes written | 6,340,340
bytes/sec | 83.434 secs
     
    Run 3:
    1,000,000 records written | 13,313 records/sec | 529,000,000 bytes written | 7,042,909
bytes/sec | 75.111 secs
    1,000,000 records written | 13,304 records/sec | 529,000,000 bytes written | 7,038,318
bytes/sec | 75.160 secs
    1,000,000 records written | 13,210 records/sec | 529,000,000 bytes written | 6,988,203
bytes/sec | 75.699 secs
    1,000,000 records written | 13,215 records/sec | 529,000,000 bytes written | 6,991,066
bytes/sec | 75.668 secs
    1,000,000 records written | 13,204 records/sec | 529,000,000 bytes written | 6,985,065
bytes/sec | 75.733 secs
     
    Run 4:
    1,000,000 records written | 13,252 records/sec | 529,000,000 bytes written | 7,010,801
bytes/sec | 75.455 secs
    1,000,000 records written | 13,133 records/sec | 529,000,000 bytes written | 6,947,454
bytes/sec | 76.143 secs
    1,000,000 records written | 13,083 records/sec | 529,000,000 bytes written | 6,921,184
bytes/sec | 76.432 secs
    1,000,000 records written | 13,095 records/sec | 529,000,000 bytes written | 6,927,529
bytes/sec | 76.362 secs
    1,000,000 records written | 13,037 records/sec | 529,000,000 bytes written | 6,896,641
bytes/sec | 76.704 secs
{noformat}

> Allow for use of hflush instead of hsync
> ----------------------------------------
>
>                 Key: ACCUMULO-2842
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2842
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: logger
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.5.2, 1.6.1, 1.7.0
>
>
> We keep getting onto the discussion on hsync versus hflush performance; however, we lack
a way to actually test this.
> Make the invocation of hflush or hsync configurable. It should not change the default,
so users on existing major releases should not be affected unless they choose to change their
configuration.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message