hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J Chouraria (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1616) hadoop fs -put and -copyFromLocal do not support globs in the source path
Date Wed, 09 Feb 2011 16:13:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992549#comment-12992549
] 

Harsh J Chouraria commented on HDFS-1616:
-----------------------------------------

This issue is a HADOOP (Common) one, not HDFS.

You can achieve a solution to globbing by Hadoop DFS by using {{FsShell.copy}} via {{hdfs
-cp}} with a command like:
{{hdfs -cp 'file:/home/test/*.jar' /destination}}

This command does a globbing on its own (not shell-driven). Should also work inside Grunt/etc.{{.}}

To programmatically call {{fs}} sub-functions, it would be wise use the Hadoop's {{FileSystem}}
API directly in your language instead. There are Python bindings for HDFS available, to start
with, apart from the Java one provided.

Regarding glob differences in Shell/Hadoop, there isn't really a standard available to conform
to (please correct me if I'm wrong). A good collection of cases is covered in {{test/o.a.h.fs.TestGlobPaths}},
which IMHO caters to most of globbing requirements (there's also a subset of regular-expression
support available).

> hadoop fs -put and -copyFromLocal do not support globs in the source path
> -------------------------------------------------------------------------
>
>                 Key: HDFS-1616
>                 URL: https://issues.apache.org/jira/browse/HDFS-1616
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>         Environment: Cloudera CDH3b3
>            Reporter: Jay Hacker
>            Priority: Minor
>
> I'd like to be able to use Hadoop globbing with the FsShell -put command, but it doesn't
work:
> {noformat}
> $ ls
> file1 file2
> $ hadoop fs -put '*' .
> put: File * does not exist.
> {noformat}
> This has probably gone unnoticed because your shell usually handles it, but a) I'd like
to be able to call 'hadoop fs' programatically without a shell, b) it doesn't work in Pig
or Grunt, where there is no shell helping you, and c) Hadoop globbing differs from shell globbing
and it would be nice to be able to use it consistently.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message