hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15460) S3A FS to add "s3a:no-existence-checks" to the builder file creation option set
Date Tue, 15 May 2018 16:29:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476095#comment-16476095

Steve Loughran commented on HADOOP-15460:

we'd skip both the checks at the beginning, and any DELETE calls put upstream at the end.
For S3Guard we still want to update the DDB tables, as long as the cost is low. Stephan is
really motivated by the problem of "writing small checkpoint files every few seconds"; there's
too much overhead around the PUT for their code righ tnow.

> S3A FS to add  "s3a:no-existence-checks" to the builder file creation option set
> --------------------------------------------------------------------------------
>                 Key: HADOOP-15460
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15460
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Major
> As promised to [~StephanEwen]: add and s3a-specific option to the builder-API to create
files for all existence checks to be skipped.
> This
> # eliminates a few hundred milliseconds
> # avoids any caching of negative HEAD/GET responses in the S3 load balancers.
> Callers will be expected to know what what they are doing.
> FWIW, we are doing some PUT calls in the committer which bypass this stuff, for the same
reason. If you've just created a directory, you know there's nothing underneath, so no need
to check.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message