hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Niels Basjes (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-7141) hadoop fs -put and -copyFromLocal do not support globs in the source path
Date Sat, 19 Feb 2011 09:27:38 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996737#comment-12996737
] 

Niels Basjes commented on HADOOP-7141:
--------------------------------------

@Harsh - Regarding your second point of accidentally matching local files that happen to exist
on HDFS with the same name: That was not the case in my example.

To illustrate this examine the following commands:

We start to check we're clean:
{code}
[niels@hacker ~/test]$ pwd
/home/niels/test
[niels@hacker ~/test]$ ls -laF testfile-*
-rw-rw-r--. 1 niels niels 12 Feb 19 10:03 testfile-1.txt
-rw-rw-r--. 1 niels niels 12 Feb 19 10:03 testfile-2.txt
-rw-rw-r--. 1 niels niels 12 Feb 19 10:03 testfile-3.txt
-rw-rw-r--. 1 niels niels 12 Feb 19 10:03 testfile-4.txt
[niels@hacker ~/test]$ ls -laF /tmp/testfile-*
ls: cannot access /tmp/testfile-*: No such file or directory
[niels@hacker ~/test]$ hadoop fs -ls /tmp/
[niels@hacker ~/test]$ hadoop fs -ls /tmp/testfile-*
ls: Cannot access /tmp/testfile-*: No such file or directory.
{code}

Now upload the files (this uses clientside bash globbing)
{code}
[niels@hacker ~/test]$ hadoop fs -put testfile-* /tmp
[niels@hacker ~/test]$ hadoop fs -ls /tmp/
Found 4 items
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-1.txt
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-2.txt
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-3.txt
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-4.txt
{code}

Hadoop fs -ls  does HDFS side globbing
{code}
[niels@hacker ~/test]$ hadoop fs -ls /tmp/testfile-*
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-1.txt
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-2.txt
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-3.txt
-rw-r--r--   1 niels supergroup         12 2011-02-19 10:21 /tmp/testfile-4.txt
{code}

Hadoop fs -rm  does HDFS side globbing
{code}
[niels@hacker ~/test]$ hadoop fs -rm /tmp/testfile-*
Deleted hdfs://localhost/tmp/testfile-1.txt
Deleted hdfs://localhost/tmp/testfile-2.txt
Deleted hdfs://localhost/tmp/testfile-3.txt
Deleted hdfs://localhost/tmp/testfile-4.txt
{code}

> hadoop fs -put and -copyFromLocal do not support globs in the source path
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7141
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>         Environment: Cloudera CDH3b3
>            Reporter: Jay Hacker
>            Priority: Minor
>
> I'd like to be able to use Hadoop globbing with the FsShell -put command, but it doesn't
work:
> {noformat}
> $ ls
> file1 file2
> $ hadoop fs -put '*' .
> put: File * does not exist.
> {noformat}
> This has probably gone unnoticed because your shell usually handles it, but a) I'd like
to be able to call 'hadoop fs' programatically without a shell, b) it doesn't work in Pig
or Grunt, where there is no shell helping you, and c) Hadoop globbing differs from shell globbing
and it would be nice to be able to use Hadoop globbing consistently throughout Hadoop.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message