hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13998) initial s3guard preview
Date Fri, 02 Jun 2017 17:23:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035077#comment-16035077

Steve Loughran commented on HADOOP-13998:

bq. We've run our standard downstream Hive, Spark, MR, Impala, scale, and performance tests

If these tests were working *before* you turned s3guard on then they weren't catching inconsistencies
& so were lucky (as mine were). I'm running my spark committer tests with the inconsistent
client turned on, and it is repeatedly failing the classic & magic committers without
s3guard enabled: both depend on consistent listing. Also found a brittleness in path cleanup
for the magic committer too; cleanup code *must* handle an FNFE if there's a file returned
in the listing but which isn't there in the GET. This is why I'd like the  factory for the
inconsistent client be in src/main: it lets anyone turn on inconsistency for their test runs

bq. This is a good point. Do you prefer timing-based microbenchmarks, or S3 request statistics

the instrumentation ones are way less brittle; Ming has been fixing some nanotimer-assertion
in WASB which was failing intermittently. I have some tests somewhere which call listFiles(recursive)
against the amazon landsat store: that's the reference example of a deep and wide directory

> initial s3guard preview
> -----------------------
>                 Key: HADOOP-13998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13998
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Steve Loughran
> JIRA to link in all the things we think are needed for a preview/merge into trunk

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message