Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 46B5F10358 for ; Wed, 7 Jan 2015 23:45:28 +0000 (UTC) Received: (qmail 25180 invoked by uid 500); 7 Jan 2015 23:45:22 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 25050 invoked by uid 500); 7 Jan 2015 23:45:22 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 25032 invoked by uid 99); 7 Jan 2015 23:45:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jan 2015 23:45:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cnauroth@hortonworks.com designates 209.85.216.45 as permitted sender) Received: from [209.85.216.45] (HELO mail-qa0-f45.google.com) (209.85.216.45) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jan 2015 23:45:17 +0000 Received: by mail-qa0-f45.google.com with SMTP id f12so5068558qad.4 for ; Wed, 07 Jan 2015 15:44:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=n9IAbl8ZJo3iV1NwcmoCAscpjoA5nvXewWItiqKSdBA=; b=RyS/98sEqabLXw/xzkpW8Z0bUf02gf1/tLmOboKF5+MYTBSWROgLCm9TDwfTMXP1X/ 8+6OCAvngR782aYNCWf2EUOBILM0AmtzbY6Qn/yD4FqkIClXMdH+Gg8zhbokvP6TXDXg T8hG8S9IiQX6h58Sg14odFpWwYZ31gS+pQuBmZAeT9CT7tygmeCIXoyKYi7avBNex4tQ a9j8dWVp8wBd3nWF2yJuEApHnhzGrtTNY8zsvxr5a1Wi6lDBvpHobmZpiP1fdl4yJ/zK mbfq2wfHVeXfj8R0Qq4Fzg9MdNP0nETjeyaONw6pQ7uKwTilnjae9SPyNYpVZvFLyYRv 0N7Q== X-Gm-Message-State: ALoCoQljEwdNM6fElFagjkD8gwySRJdZqg6qHFqECdfzwzAgXIksNXMI/gCUaNm4QrHARKQnwPx7KhVkak6CKzCsEVuJDUXdWcc0adfT6WbOb9hH6mYPW7I= MIME-Version: 1.0 X-Received: by 10.224.54.2 with SMTP id o2mr9475005qag.63.1420674250401; Wed, 07 Jan 2015 15:44:10 -0800 (PST) Received: by 10.96.66.67 with HTTP; Wed, 7 Jan 2015 15:44:10 -0800 (PST) In-Reply-To: <54AA2A9E.7050105@perfect-cn.cn> References: <54AA2A9E.7050105@perfect-cn.cn> Date: Wed, 7 Jan 2015 15:44:10 -0800 Message-ID: Subject: Re: Putting multiple files.. From: Chris Nauroth To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1132f080098466050c1880a5 X-Virus-Checked: Checked by ClamAV on apache.org --001a1132f080098466050c1880a5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Also, bash wildcard expansion should automatically put the full list of matching files into the list of local source arguments prior to execution. For example, assuming 3 files named hello1, hello2 and hello3, then running the following command... hdfs dfs -put hello* /user/chris ...should turn into this automatically... hdfs dfs -put hello1 hello2 hello3 /user/chris If that meets your need, then it avoids the need to spell out each local source file name individually. Chris Nauroth Hortonworks http://hortonworks.com/ On Sun, Jan 4, 2015 at 10:09 PM, "Cao Yi.=E6=9B=B9=E9=93=B1" wrote: > At the current version(2.6.0), *hadoop fs* is deprecated, use *hdfs dfs* > instead. > > Ref: > > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/C= ommandsManual.html > > =E5=9C=A8 2014=E5=B9=B412=E6=9C=8829=E6=97=A5 05:26, Abhishek Singh =E5= =86=99=E9=81=93: > > Hello Anil, > > There are 2 ways I'm aware of :- > > 1) use put command > > put > > Usage: hadoop fs -put ... > > Copy single src, or multiple srcs from local file system to the > destination filesystem. Also reads input from stdin and writes to > destination filesystem. > > hadoop fs -put localfile /user/hadoop/hadoopfile > hadoop fs -put localfile1 localfile2 /user/hadoop/hadoopdir > hadoop fs -put localfile hdfs://nn.example.com/hadoop/hadoopfile > hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopfile > Reads the input from stdin. > > Exit Code: > > Returns 0 on success and -1 on error. > > 2) Create a shell script for your custom need. > > To give you a vague idea here's one of the link on stackoverflow which is > similar to what you are demanding:- > > > http://stackoverflow.com/questions/12790166/shell-script-to-move-files-in= to-a-hadoop-cluster > > Please reach out for further discussion! > > Thanks! > > Regards, > > Abhishek Singh > On Dec 28, 2014 3:52 AM, "Anil Jagtap" wrote: > >> Dear All, >> >> Just wanted to know if there is a way to copy multiple files using >> hadoop fs -put. >> >> Instead of specifying individual name I provide wild-chars and >> respective files should get copied. >> >> Thank You. >> >> Rgds, Anil >> > > -- > Best Regards, > Cao Yi, =E6=9B=B9=E9=93=B1 > Tel: 189-8052-8753 > > =E5=8C=97=E4=BA=AC=E6=99=AE=E8=8F=B2=E7=89=B9=E5=B9=BF=E5=91=8A=E6=9C=89= =E9=99=90=E5=85=AC=E5=8F=B8=EF=BC=88=E6=88=90=E9=83=BD=EF=BC=89 > > --=20 CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to= =20 which it is addressed and may contain information that is confidential,=20 privileged and exempt from disclosure under applicable law. If the reader= =20 of this message is not the intended recipient, you are hereby notified that= =20 any printing, copying, dissemination, distribution, disclosure or=20 forwarding of this communication is strictly prohibited. If you have=20 received this communication in error, please contact the sender immediately= =20 and delete it from your system. Thank You. --001a1132f080098466050c1880a5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Also, bash wildcard expansion should automatically put the= full list of matching files into the list of local source arguments prior = to execution.=C2=A0 For example, assuming 3 files named hello1, hello2 and = hello3, then running the following command...

hdfs dfs -= put hello* /user/chris

...should turn into this au= tomatically...

hdfs dfs -put hello1 hello2 hello3 = /user/chris

If that meets your need, then it avoid= s the need to spell out each local source file name individually.

Chris Nauroth
Hortonworks
<= a href=3D"http://hortonworks.com/" target=3D"_blank">http://hortonworks.com= /


On Sun, Jan 4, 2015 at 10:09 PM, "Cao Y= i.=E6=9B=B9=E9=93=B1" <caoyi@perfect-cn.cn> wrote:
=20 =20 =20
At the current version(2.6.0), hadoop fs is deprecated, use hdfs dfs instead.

Ref:
http://hadoop.apache.org/doc= s/current/hadoop-project-dist/hadoop-common/CommandsManual.html

=E5=9C=A8 2014=E5=B9=B412=E6=9C=8829=E6=97=A5 05:26, Abhishek Sing= h =E5=86=99=E9=81=93:

Hello Anil,

There are 2 ways I'm aware of :-

1) use put command

put

Usage: hadoop fs -put <localsrc> ... <dst>

Copy single src, or multiple srcs from local file system to the destination filesystem. Also reads input from stdin and writes to destination filesystem.

=C2=A0=C2=A0=C2=A0 hadoop fs -put localfile /user/hado= op/hadoopfile
=C2=A0=C2=A0=C2=A0 hadoop fs -put localfile1 localfile2 /user/hadoo= p/hadoopdir
=C2=A0=C2=A0=C2=A0 hadoop fs -put localfile hdfs://nn.example.com/hadoop= /hadoopfile
=C2=A0=C2=A0=C2=A0 hadoop fs -put - hdfs://nn.example.com/hadoop/hadoopf= ile
=C2=A0=C2=A0=C2=A0 Reads the input from stdin.

Exit Code:

Returns 0 on success and -1 on error.

2) Create a shell script for your custom need.

To give you a vague idea here's one of the link on stackoverflow which is similar to what you are demanding:-

http:/= /stackoverflow.com/questions/12790166/shell-script-to-move-files-into-a-had= oop-cluster

Please reach out for further discussion!

Thanks!

Regards,

Abhishek Singh

On Dec 28, 2014 3:52 AM, "Anil Jagtap= " <anil= .jagtap@gmail.com> wrote:
Dear All,

Just wanted to know if there is a way to copy multiple files using hadoop fs -put.

Instead of specifying individual name I provide wild-chars and respective files should get copied.

Thank You.

Rgds, Anil

--=20
Best Regards,
Cao Yi, =E6=9B=B9=E9=93=B1
Tel: 189-8052-8753

=E5=8C=97=E4=BA=AC=E6=99=AE=E8=8F=B2=E7=89=B9=E5=B9=BF=E5=91=8A=E6=9C=89=E9=
=99=90=E5=85=AC=E5=8F=B8=EF=BC=88=E6=88=90=E9=83=BD=EF=BC=89


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --001a1132f080098466050c1880a5--