hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Hitchcock (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5635) distributed cache doesn't work with other distributed file systems
Date Tue, 07 Apr 2009 21:42:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696765#action_12696765

Andrew Hitchcock commented on HADOOP-5635:

It sounds like what you want is a new featue whereas this patch is just to fix a bug.

Currently the behavior is not right. If a user specifies a non-HDFS URI for distributed cache
then the job will fail because the tasks look for the file in HDFS. This patch fixes that
for cases when the user specifies a URI to another distributed file system. With the patch,
if a user specifies KFS or S3N (and the file system is properly configured) then the job will
succeed. The behavior for specifying a URI not accessible on every machine remains unchanged:
the job will fail as tasks are unable to reach the URI.

I think a feature for administrators to restrict distributed cache access to certain file
systems should be a new Jira.

> distributed cache doesn't work with other distributed file systems
> ------------------------------------------------------------------
>                 Key: HADOOP-5635
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5635
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: filecache
>            Reporter: Andrew Hitchcock
>            Priority: Minor
>         Attachments: fix-distributed-cache.patch
> Currently the DistributedCache does a check to see if the file to be included is an HDFS
URI. If the URI isn't in HDFS, it returns the default filesystem. This prevents using other
distributed file systems -- such as s3, s3n, or kfs  -- with distributed cache. When a user
tries to use one of those filesystems, it reports an error that it can't find the path in

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message