hadoop fs -cp s3n://acessKey:acessSecret@bucket.= name/srcData&quo= t; /test/

Or w= ith distcp:=C2=A0

hadoop distcp=C2=A0s3n://acessKey:acessSecret@bucket.name/srcData" /test/

HTH.=C2=A0
=

Shumin

On Wed, Mar 6, 2013 at 5:44 AM, Subroto <ssanyal@datameer.com> wrote:

Hi Mike,

I have trie= s distcp as well and it ended up with exception:
13/03/06 05= :41:13 INFO tools.DistCp: srcPaths=3D[s3n://acessKey:acessSecret@dm.test= .bucket/srcData]

13/03/06 05:41:13 INFO tools.DistCp: destPath=3D/test/srcData
13/03/06 05:41:18 INFO tools.DistCp: /test/srcData does not exist.
<= div>org.apache.hadoop.tools.DistCp$DuplicationException: Invalid input, the= re are duplicated files in the sources: s3n://acessKey:acessSecret@dm.te= st.bucket/srcData/compressed, s3n://acessKey:acessSecret@dm.test.buc= ket/srcData/compressed

at org.apache.hadoop.tool= s.DistCp.checkDuplication(DistCp.java:1368)
at org.apache.hadoop.tools.DistCp.setup(DistCp.java= :1176)

at org.apache.hadoop.tool= s.DistCp.copy(DistCp.java:666)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.Tool= Runner.run(ToolRunner.java:65)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

at org.apache.hadoop.tool= s.DistCp.main(DistCp.java:908)

One more interestin= g stuff to notice is that same thing works nicely with hadoop 2.0

Cheers,

Subroto Sanyal

On Mar 6, 2013, at 11:12 AM, Michel Segel wrot= e:

Have you tried= using distcp?

Sent from a remote device. Please excuse any typos...

Mike Segel

On Mar 5, 2013, at 8:37 AM, Subroto <ssanyal@datameer.co= m> wrote:

Hi,

Its not bec= ause there are too many recursive folders in S3 bucket; in-fact there is no= recursive folder in the source.
If I list the S3 bucket with Nat= ive S3 tools I can find a file srcData with size 0 in the folder srcData.= =C2=A0

The copy command keeps on creating folder=C2=A0=C2=A0/test/srcData/src= Data/srcData (keep on appending srcData).

Cheers,<= /div>
Subroto Sanyal

On Mar 5, 2013, at 3:32 PM= , =E5=8D=96=E6=8A=A5=E7=9A=84=E5=B0=8F=E8=A1=8C=E5=AE=B6 wrote:

Hi Subroto,

I didn't use the s3n filesystem.But =C2=A0from the output "cp: java.io.IOException: mkdirs: Pathname too lo= ng.=C2=A0 Limit 8000 characters, 1000 levels.", I think this is because the problem of the path. Is the pa= th longer than 8000 characters or the level is more than 1000?

You only have 998 folders.Maybe the la= st one is more than 8000 characters.Why not count the last one's length= ?

BRs//Julian

<= /span>

=

= ------------------=C2=A0Original=C2=A0------------------
From: =C2=A0&qu= ot;Subroto"<ssanyal@datameer.com>;

Date: =C2=A0Tue, Mar 5, 2013 10:22 PM
To: =C2= =A0"user"<user@hadoop.apache.org>;
Sub= ject: =C2=A0S3N copy creating recursive folders

Hi,

I am using Hadoop 1.0.3 and trying to execu= te:
hadoop fs -cp s3n://acessKey:acessSecret@bucket.name/srcData&= quot; /test/srcData

This ends up with:
cp: java.io.IOException: m= kdirs: Pathname too long.=C2=A0 Limit 8000 characters, 1000 levels.

When I try to list the folder recursively /test/srcData: it lists 998 f= olders like:
drwxr-xr-x=C2=A0=C2=A0 - root supergroup=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 2013-03-05 08:49 /test/srcData/srcDa= ta
drwxr-xr-x=C2=A0=C2=A0 - root supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0 2013-03-05 08:49 /test/srcData/srcData/srcDat= a
drwxr-xr-x=C2=A0=C2=A0 - root supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0 2013-03-05 08:49 /test/srcData/srcData/srcData/srcD= ata
drwxr-xr-x=C2=A0=C2=A0 - root supergroup=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0 2013-03-05 08:49 /test/srcData/srcData/srcDat= a/srcData/srcData
drwxr-xr-x=C2=A0=C2=A0 - root supergroup=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 2013-03-05 08:49 /test/srcData= /srcData/srcData/srcData/srcData/srcData

Is there a problem with s3n filesystem ??

Cheers,
Subroto San= yal