hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14467) S3Guard: Improve FNFE message when opening a stream
Date Wed, 31 May 2017 11:28:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031008#comment-16031008

Steve Loughran commented on HADOOP-14467:

likely root cause would be that DDB is inconsistent with the FS.

# I think we should add a wiki entry pointing to a (new) hadoop cwiki entry. I say cwiki as
the older wiki may be phased out at some point.
# maybe also the input stream could differentiate from "never opened the file" from "opened,
it was there, now it isnt". which can happen if someone deletes it during the read process

> S3Guard: Improve FNFE message when opening a stream
> ---------------------------------------------------
>                 Key: HADOOP-14467
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14467
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
> Following up on the [discussion on HADOOP-13345|https://issues.apache.org/jira/browse/HADOOP-13345?focusedCommentId=16030050&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16030050],
because S3Guard can serve getFileStatus() from the MetadataStore without doing a HEAD on S3,
a FileNotFound error on a file due to S3 GET inconsistency does not happen on open(), but
on the first read of the stream.  We may add retries to the S3 client in the future, but for
now we should have an exception message that indicates this may be due to inconsistency (assuming
it isn't a more straightforward case like someone deleting the object out from under you).
> This is expected to be a rare case, since the S3 service is now mostly consistent for

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message