Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 7 Jul 2017 02:44:00 +0000 (UTC)
From: "Aaron Fabbri (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13075981.1496178543000.190373.1499395440242@Atlassian.JIRA>
In-Reply-To: <JIRA.13075981.1496178543000@Atlassian.JIRA>
References: <JIRA.13075981.1496178543000@Atlassian.JIRA> <JIRA.13075981.1496178543453@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14468) S3Guard: make short-circuit
 getFileStatus() configurable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 07 Jul 2017 02:44:04 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077499#comment-16077499 ] 

Aaron Fabbri commented on HADOOP-14468:
---------------------------------------

{quote}
That said, looking at all the places we call getFileStatus, it'd be a useful little sanity check all round.
{quote}
Yeah. It would be interesting to collect statistics on long-running clusters on how often inconsistency happens.

Sounds like we're ok with the behavior of failing after open().  Your example of deleted file or inconsistency causing similar behavior is a good point.  I'll leave this as minor priority for now and focus on HADOOP-14467 first.

> S3Guard: make short-circuit getFileStatus() configurable
> --------------------------------------------------------
>
>                 Key: HADOOP-14468
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14468
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>            Priority: Minor
>
> Currently, when S3Guard is enabled, getFileStatus() will skip S3 if it gets a result from the MetadataStore (e.g. dynamodb) first.
> I would like to add a new parameter {{fs.s3a.metadatastore.getfilestatus.authoritative}} which, when true, keeps the current behavior.  When false, S3AFileSystem will check both S3 and the MetadataStore.
> I'm not sure yet if we want to have this behavior the same for all callers of getFileStatus(), or if we only want to check both S3 and MetadataStore for some internal callers such as open().


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org