hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14633) S3Guard: optimize create codepath
Date Sat, 08 Jul 2017 01:17:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Aaron Fabbri updated HADOOP-14633:
    Attachment: HADOOP-14633-HADOOP-13345.001.patch

Attaching v1 patch that gives almost 2x speedup for pathological create workload.

- Adds new scale test for create performance, using a pathological case to stress S3guard
ancestor creation.
- Makes changes suggested in my comment linked in JIRA description.

Performance tests in US West 2 from home laptop (low latency to datacenter actually):

500 files of depth 10 via ITestS3ACreatePerformance

w/o s3guard, in *msec per create*:
1.92 IOPS

- w/ s3guard, dynamo:
0.87 IOPS

w/ s3guard, dynamo + this patch:
1.56 IOPS

Currently running full suite of integration tests.. but wanted to post this before leaving
for day.. Will followup with integration test results.

> S3Guard: optimize create codepath
> ---------------------------------
>                 Key: HADOOP-14633
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14633
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>         Environment: 
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>            Priority: Minor
>         Attachments: HADOOP-14633-HADOOP-13345.001.patch
> Following up on HADOOP-14457, a couple of things to do that will improve create performance
as I mentioned in the comment [here|https://issues.apache.org/jira/browse/HADOOP-14457?focusedCommentId=16078465&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078465]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message