hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Da Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15663) ABFS: Simplify configuration
Date Wed, 29 Aug 2018 20:52:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596843#comment-16596843

Da Zhou commented on HADOOP-15663:

Thanks for the feedback.
Attaching patch: HADOOP-15663-HADOOP-15407-003.patch: 
-  updated the testing doc.
-  fixed typos and  updated logger in AbfsTestUtils
-  removed "fs.AbstractFileSystem.wasb.impl " and " fs.azure.scale.test.enabled" from config
-  added "fs.AbstractFileSystem.wasb.impl" and "fs.AbstractFileSystem.wasbs.impl" into core-default.xml.

> ABFS: Simplify configuration
> ----------------------------
>                 Key: HADOOP-15663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15663
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>            Reporter: Thomas Marquardt
>            Assignee: Da Zhou
>            Priority: Major
>         Attachments: HADOOP-15663-HADOOP-15407-001.patch, HADOOP-15663-HADOOP-15407-002.patch,
HADOOP-15663-HADOOP-15407-003.patch, HADOOP-15663-HADOOP-15407-004.patch
> Configuration for WASB and ABFS is too complex.  The current approach is to use four
files for test configuration. 
> Both WASB and ABFS have basic test configuration which is committed to the repo (azure-test.xml
and azure-bfs-test.xml).  Currently these contain the fs.AbstractFileSystem.[scheme].impl
configuration, but otherwise are empty except for an include reference to a file containing
the endpoint credentials. 
> Both WASB and ABFS have endpoint credential configuration files (azure-auth-keys.xml
and azure-bfs-auth-keys.xml).  These have been added to .gitignore to prevent them from
accidentally being submitted in a patch, which would leak the developers storage account credentials. 
These files contain account names, storage account keys, and service endpoints.
> There is some overlap of the configuration for WASB and ABFS, where they use the same
property name but use different values.  
> 1) Let's reduce the number of test configuration files to one, if possible.
> 2) Let's simplify the account name, key, and endpoint configuration for WASB and ABFS
if possible, but still support the legacy way of doing it, which is very error prone.
> 3) Let's improve error handling, so that typos or misconfiguration are not so difficult
to troubleshoot.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message