hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4453) Improve ssl handling for distcp
Date Wed, 22 Oct 2008 22:51:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641990#action_12641990

Chris Douglas commented on HADOOP-4453:

bq. SSL properties are system properties that affect all SSL connections that the client make,
not just those used in HsftpFileSystem. 

This is a good point, but it also highlights the extent of this change. Since the purpose
of this JIRA is to improve the existing ssl support for HsftpFileSystem, its side-effects
shouldn't extend to all map/reduce tasks. Ideally, it should also maintain distcp's FileSystem

The server code looks good. The client code in Child and DistCp can probably be moved into
HsftpFileSystem::initialize, which should also warn if it's called more than once in the same
JVM (assuming a second HsftpFileSystem handle can invalidate the first; FileSystem cache hits
should be benign). Unfortunately, this creates a dilemma for distcp: its client and task resources
may be at different locations, but HsftpFileSystem will use the same property. This is going
to be true for many users of HsftpFileSystem.

Lacking a general need for similar, asymmetric configuration, I'd propose adding a static
config method to HsftpFileSystem:

public static void setSslConfigLocation(String loc, Configuration conf) {
  conf.set("dfs.https.client.keystore.resource", loc);

And calling this before submitting the job from DistCp, if {{\-mapredSslLoc <loc>}}
is specified. If there is a general need, we can look into extending GenericOptionsParser
to support something more clever.

> Improve ssl handling for distcp
> -------------------------------
>                 Key: HADOOP-4453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4453
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.20.0
>            Reporter: Chris Douglas
>            Assignee: Kan Zhang
>             Fix For: 0.20.0
>         Attachments: 4453-0.patch, 4453-2.patch
> HsftpFileSystem is an ad hoc way to read from HDFS over ssl, targeting distcp. Its organization
can be improved and its support of ssl features expanded.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message