hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12353) S3 Native filesystem does not retry all connection failures
Date Thu, 27 Aug 2015 10:21:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716432#comment-14716432

Steve Loughran commented on HADOOP-12353:

I'm happy to have better connectivity handling, but worried that this patch  is changing retry
policy handling from explicit type match to subclass --with implications for all classes setting
up the retry policy. 

I'd thought initially that re-ordering definitions of policies such that most-generic comes
last would suffice, but it won't as the order of {{Map.keySet()}} is undefined.

Someone who knows the retry logic needs to review this. At the very least, I'd retain the
initial map lookup, and only if there was no match fall back to the keyset scan. That way,
explicit policies are guaranteed to get picked up first

> S3 Native filesystem does not retry all connection failures
> -----------------------------------------------------------
>                 Key: HADOOP-12353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.7.1
>            Reporter: Mariusz Strzelecki
>            Assignee: Mariusz Strzelecki
>            Priority: Minor
>         Attachments: HADOOP-12353.001.patch, HADOOP-12353.002.patch
> Current implementation of NativeS3FileSystem.java uses RetryProxy that retries exceptions
that may occur on network communication with S3 API, but these exceptions must be exact instances
of IOException:
> https://github.com/apache/hadoop/blob/master/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/NativeS3FileSystem.java#L349
> Our tests show that HttpClient throws IOException subclasses which are not handled by
> Additionally, not all methods that call S3 API are listed to be handled, i.e. storeEmptyFile
and retrieveMetadata are missing.

This message was sent by Atlassian JIRA

View raw message