hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14138) Remove S3A ref from META-INF service discovery, rely on existing core-default entry
Date Wed, 05 Apr 2017 17:18:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957261#comment-15957261
] 

Siddharth Seth commented on HADOOP-14138:
-----------------------------------------

[~steve_l] - why should s3a entries exist in core-default.xml?
core-default is supposed to contain defaults for most config values, and serves as documentation.

If someone wants to use s3a, I'd expect them to explicitly set it up in their Configuration,
or rely on the ServiceLoader approach - which this jira reverses. 

> Remove S3A ref from META-INF service discovery, rely on existing core-default entry
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-14138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14138
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Critical
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha3
>
>         Attachments: HADOOP-14138.001.patch, HADOOP-14138-branch-2-001.patch
>
>
> As discussed in HADOOP-14132, the shaded AWS library is killing performance starting
all hadoop operations, due to classloading on FS service discovery.
> This is despite the fact that there is an entry for fs.s3a.impl in core-default.xml,
*we don't need service discovery here*
> Proposed:
> # cut the entry from {{/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem}}
> # when HADOOP-14132 is in, move to that, including declaring an XML file exclusively
for s3a entries
> I want this one in first as its a major performance regression, and one we coula actually
backport to 2.7.x, just to improve load time slightly there too



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message