hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-13934) Multipart uploaders to be created through API call to FileSystem/FileContext, not service loader
Date Sun, 23 Sep 2018 11:56:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HDFS-13934:
----------------------------------
    Description: 
the Multipart Uploaders are created via service loaders. This is troublesome

# HADOOP-12636, HADOOP-13323, HADOOP-13625 highlight how the load process forces the transient
loading of dependencies.  If a dependent class cannot be loaded (e.g aws-sdk is not on the
classpath), that service won't load. Without error handling round the load process, this stops
any uploader from loading. Even with that error handling, the performance hit of that load,
especially with reshaded dependencies, hurts performance (HADOOP-13138).
# it makes wrapping the the load with any filter impossible, stops transitive binding through
viewFS, mocking, etc.
# It complicates security in a kerberized world. If you have an FS instance of user A, then
you should be able to create an MPU instance with that user's permissions. currently, if a
service were to try to create one, you'd be looking at doAs() games around the service loading,
and a more complex bind process.

Proposed
# remove the service loader mech entirely
# add to FS & FC as createMultipartUploader(path) call, which will create one bound to
the current FS, with its permissions, DTs, etc.


  was:
the Multipart Uploaders are created via service loaders. This is troublesome

# HADOOP-12636, HADOOP-13323, HADOOP-13625 highlight how the load process forces the transient
loading of dependencies.  If a dependent class cannot be loaded (e.g aws-sdk is not on the
classpath), that service won't load. Without error handling round the load process, this stops
any uploader from loading. Even with that error handling, the performance hit of that load,
especially with reshaded dependencies, hurts performance (HADOOP-13138).
# it makes wrapping the the load with any filter impossible, stops transitive binding through
viewFS 
# It complicates security in a kerberized world. If you have an FS instance of user A, then
you should be able to create an MPU instance with that user's permissions. currently, if a
service were to try to create one, you'd be looking at doAs() games around the service loading,
and a more complex bind process.

Proposed
# remove the service loader mech entirely
# add to FS & FC as createMultipartUploader(path) call, which will create one bound to
the current FS, with its permissions, DTs, etc.



> Multipart uploaders to be created through API call to FileSystem/FileContext, not service
loader
> ------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-13934
>                 URL: https://issues.apache.org/jira/browse/HDFS-13934
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: fs, fs/s3, hdfs
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Priority: Major
>
> the Multipart Uploaders are created via service loaders. This is troublesome
> # HADOOP-12636, HADOOP-13323, HADOOP-13625 highlight how the load process forces the
transient loading of dependencies.  If a dependent class cannot be loaded (e.g aws-sdk is
not on the classpath), that service won't load. Without error handling round the load process,
this stops any uploader from loading. Even with that error handling, the performance hit of
that load, especially with reshaded dependencies, hurts performance (HADOOP-13138).
> # it makes wrapping the the load with any filter impossible, stops transitive binding
through viewFS, mocking, etc.
> # It complicates security in a kerberized world. If you have an FS instance of user A,
then you should be able to create an MPU instance with that user's permissions. currently,
if a service were to try to create one, you'd be looking at doAs() games around the service
loading, and a more complex bind process.
> Proposed
> # remove the service loader mech entirely
> # add to FS & FC as createMultipartUploader(path) call, which will create one bound
to the current FS, with its permissions, DTs, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message