hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14138) Remove S3A ref from META-INF service discovery, rely on existing core-default entry
Date Wed, 19 Apr 2017 20:42:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975477#comment-15975477
] 

Siddharth Seth commented on HADOOP-14138:
-----------------------------------------

Not reading core-default was a bug, and is fixed. core-site.xml is independent of what is
in core-default.xml, if the code defaults were used.

See: HDFS-4820, HADOOP-7956. While searching for these bugs, there's several others which
relate to removing deprecated entries, fixing default values etc from *-default files (This
was within the HDFS project itself).
My concern is that an explicit requirement on the *-default files is getting introduced, while
the files should be removed IMO.

I get the problem with the serviceloaders, and the additional time introduced. Thoughts there
was an alternate plan to reduce this cost, while retaining service loaders.

Could you please elaborate on "How to pick up credentials" - and why that needs to be part
of core-default? (And not a conf.get(PARAMETER, DEFAULT)). 

> Remove S3A ref from META-INF service discovery, rely on existing core-default entry
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-14138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14138
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Critical
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3
>
>         Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance starting
all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in core-default.xml,
*we don't need service discovery here*
> Proposed:
> # cut the entry from {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file exclusively
for s3a entries
> I want this one in first as its a major performance regression, and one we coula actually
backport to 2.7.x, just to improve load time slightly there too



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message