hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-17498) Add a MkdirOperation for chained S3 operations during mkdir
Date Mon, 02 Aug 2021 15:12:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-17498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran resolved HADOOP-17498.
    Fix Version/s: 3.3.2
       Resolution: Duplicate

> Add a MkdirOperation for chained S3 operations during mkdir
> -----------------------------------------------------------
>                 Key: HADOOP-17498
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17498
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.3.2
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
> on S3A mkdirs is implemented as a walk up the tree looking for any parent dir, exiting
fast when found; failing if a file is found.
> Proposed: pull this out into its own MkDir operation with the goals of efficiency, track
duration and isolate/test
> * specific callbacks to getFileStatus and  createFakeDirectory(key);
> * Move from simple getFileStatus call to probes assuming dir exists before looking for
a file: HEAD + path /, LIST path,
> * the probe for a file  HEAD path will only succeed in a failure mode where higher cost
is a detail
> I think I'd also like to start looking at what we can do with some context tracking/telemetry
where something provides an operation ID and ability to attach this to the requests

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

View raw message