hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12667) s3a: Support createNonRecursive API
Date Tue, 22 Dec 2015 20:37:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068697#comment-15068697

Sean Mackrory commented on HADOOP-12667:

HBase's use of this API is indeed what brought the lack of support to my attention. I wouldn't
say I'm specifically trying to support HBase-on-S3, though - just trying to improve support
in general where we can. If there's no reasonable solution, there's no reasonable solution.
I certainly don't want to be increasing the potential for inconsistency if that's what it
would take.

On top of what Steve said, though, I think it's well-understood that when using Hadoop-on-S3
in general there's been the potential for consistency issues. That's improved dramatically
on S3's end over time, of course. In this case all nodes will currently always fail because
of an unsupported API. Due to the potential for inconsistency, if this API was supported,
it would be possible for some nodes to fail because of a FileAlreadyExistsException, and the
operation could be retried. That's certainly an improvement, and well within the expectations
set by the S3 filesystems, IMO. I would still vote to move in this direction.

{quote}Sean: fancy taking that on?{quote}

Yeah I think that'd be good to do. I'll wait until we reach some consensus on the consistency
issue, though.

> s3a: Support createNonRecursive API
> -----------------------------------
>                 Key: HADOOP-12667
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12667
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>            Reporter: Sean Mackrory
>            Assignee: Sean Mackrory
>         Attachments: HADOOP-12667.001.patch
> HBase and other clients rely on the createNonRecursive API, which was recently un-deprecated.
S3A currently does not support it.

This message was sent by Atlassian JIRA

View raw message