hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4453) Improve ssl handling for distcp
Date Wed, 22 Oct 2008 22:51:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641990#action_12641990
] 

Chris Douglas commented on HADOOP-4453:
---------------------------------------

bq. SSL properties are system properties that affect all SSL connections that the client make,
not just those used in HsftpFileSystem. 

This is a good point, but it also highlights the extent of this change. Since the purpose
of this JIRA is to improve the existing ssl support for HsftpFileSystem, its side-effects
shouldn't extend to all map/reduce tasks. Ideally, it should also maintain distcp's FileSystem
agnosticism.

The server code looks good. The client code in Child and DistCp can probably be moved into
HsftpFileSystem::initialize, which should also warn if it's called more than once in the same
JVM (assuming a second HsftpFileSystem handle can invalidate the first; FileSystem cache hits
should be benign). Unfortunately, this creates a dilemma for distcp: its client and task resources
may be at different locations, but HsftpFileSystem will use the same property. This is going
to be true for many users of HsftpFileSystem.

Lacking a general need for similar, asymmetric configuration, I'd propose adding a static
config method to HsftpFileSystem:
{code}

public static void setSslConfigLocation(String loc, Configuration conf) {
  conf.set("dfs.https.client.keystore.resource", loc);
}
{code}

And calling this before submitting the job from DistCp, if {{\-mapredSslLoc <loc>}}
is specified. If there is a general need, we can look into extending GenericOptionsParser
to support something more clever.

> Improve ssl handling for distcp
> -------------------------------
>
>                 Key: HADOOP-4453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4453
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.20.0
>            Reporter: Chris Douglas
>            Assignee: Kan Zhang
>             Fix For: 0.20.0
>
>         Attachments: 4453-0.patch, 4453-2.patch
>
>
> HsftpFileSystem is an ad hoc way to read from HDFS over ssl, targeting distcp. Its organization
can be improved and its support of ssl features expanded.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message