hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14633) S3Guard: optimize create codepath
Date Sat, 08 Jul 2017 01:23:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078883#comment-16078883
] 

Aaron Fabbri edited comment on HADOOP-14633 at 7/8/17 1:22 AM:
---------------------------------------------------------------

Attaching v1 patch that gives almost 2x speedup for pathological create workload.

- Adds new scale test for create performance, using a pathological case to stress S3guard
ancestor creation.
- Makes changes suggested in my comment linked in JIRA description.

Performance tests in US West 2 from home laptop (decent latency to datacenter actually):

500 files of depth 10 via ITestS3ACreatePerformance

{noformat}
w/o s3guard, in *msec per create*:
521426
1.92 IOPS

w/ s3guard, dynamo:
1153689
0.87 IOPS

w/ s3guard, dynamo + this patch:
638544
1.56 IOPS
{noformat}

Currently running full suite of integration tests.. but wanted to post this before leaving
for day.. Will followup with integration test results.



was (Author: fabbri):
Attaching v1 patch that gives almost 2x speedup for pathological create workload.

- Adds new scale test for create performance, using a pathological case to stress S3guard
ancestor creation.
- Makes changes suggested in my comment linked in JIRA description.

Performance tests in US West 2 from home laptop (low latency to datacenter actually):

500 files of depth 10 via ITestS3ACreatePerformance

{noformat}
w/o s3guard, in *msec per create*:
521426
1.92 IOPS

- w/ s3guard, dynamo:
1153689
0.87 IOPS

w/ s3guard, dynamo + this patch:
638544
1.56 IOPS
{noformat}

Currently running full suite of integration tests.. but wanted to post this before leaving
for day.. Will followup with integration test results.


> S3Guard: optimize create codepath
> ---------------------------------
>
>                 Key: HADOOP-14633
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14633
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>         Environment: 
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>            Priority: Minor
>             Fix For: HADOOP-13345
>
>         Attachments: HADOOP-14633-HADOOP-13345.001.patch
>
>
> Following up on HADOOP-14457, a couple of things to do that will improve create performance
as I mentioned in the comment [here|https://issues.apache.org/jira/browse/HADOOP-14457?focusedCommentId=16078465&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078465]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message