hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Zhuge (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14132) Filesystem discovery to stop loading implementation classes
Date Fri, 03 Mar 2017 03:21:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893664#comment-15893664
] 

John Zhuge commented on HADOOP-14132:
-------------------------------------

There are FS service loader metafiles for common, hdfs, aws, wasb, and swift.

For adls, just abandon HADOOP-14123.
For wasb, hdfs, and common, shall we create JIRAs similar to HADOOP-14138?
For swift, revert HADOOP-13606?


> Filesystem discovery to stop loading implementation classes
> -----------------------------------------------------------
>
>                 Key: HADOOP-14132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14132
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, fs/adl, fs/azure, fs/oss, fs/s3, fs/swift
>    Affects Versions: 2.7.3
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>
> Integration testing of Hadoop with the HADOOP-14040 has shown up that the move to a shaded
AWS JAR is slowing all hadoop client code down.
> I believe this is due to how we use service discovery to identify FS implementations:
the implementation classes themselves are instantiated.
> This has known problems today with classloading, but clearly impacts performance too,
especially with complex transitive dependencies unique to the loaded class.
> Proposed: have lightweight service declaration classes which implement an interface declaring
> # schema
> # classname of FileSystem impl
> # classname of AbstractFS impl
> # homepage (for third party code, support, etc)
> These are what we register and scan in the FS to look for services.
> This will leave the question about what to do for existing filesystems? I think we'll
need to retain the old code for external ones, while moving the hadoop modules to the new
ones



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message