hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14547) [WASB] the configured retry policy is not used for all storage operations.
Date Mon, 19 Jun 2017 23:17:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Thomas updated HADOOP-14547:
    Attachment: HADOOP-14547.001.patch

Attaching patch HADOOP-14547.001.patch.

The issue is fixed by reading the WASB configuration settings before creating and initializing
the storage components (CloudBlobClient, CloudBlobContainer, etc).  The retry policy and session
timeout are now stored by the StorageInterface implementations and used to properly initialize
the storage objects.

Unfortunately, the retry policy cannot easily be validated with unit tests due to functionality
that is not exposed by the Azure Storage SDK.  I have carefully validated this change in the
debugger to ensure that the user configured retry policy is used.

> [WASB] the configured retry policy is not used for all storage operations.
> --------------------------------------------------------------------------
>                 Key: HADOOP-14547
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14547
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/azure
>    Affects Versions: 2.9.0, 3.0.0-alpha4
>            Reporter: Thomas
>            Assignee: Thomas
>             Fix For: 2.9.0, 3.0.0-alpha4
>         Attachments: HADOOP-14547.001.patch
> There are a few places where the WASB retry policy was not used, and instead the Azure
Storage SDK default retry policy was used.  These places include some calls to blob.exist(),
container.exists(), blob.delete(), and operations with secure mode enabled (fs.azure.secure.mode
= true).  
> You can set a break point on com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry
in the Azure Storage SDK and see that the WASB configured retry policy (DEFAULT_MAX_RETRY_ATTEMPTS,
etc.) is not used.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message