hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: distcp fails with s3n or s3a in 2.6.0
Date Tue, 12 May 2015 04:59:39 GMT
Hello Steve,

There was a similar discussion about this on the mailing list last month:


I recommend reading through all of the information in that thread and implementing the suggestions.
 You'll also see reference to a jira issue that tracks improvements we need to make in the
documentation to make this more obvious.  Feel free to watch that issue if you'd like to receive
status updates as the documentation progresses.

I hope this helps.

--Chris Nauroth

From: Stephen Armstrong <stephen.armstrong@linqia.com<mailto:stephen.armstrong@linqia.com>>
Reply-To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Date: Friday, May 8, 2015 at 3:33 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: distcp fails with s3n or s3a in 2.6.0


I've been stuck on a classpath problem for a while now, and am hoping for some help. Hadoop
2.6.0 no longer seems to support distcp to S3 for me. I posted this on Stack Overflow (http://stackoverflow.com/questions/30108760/s3n-and-s3a-distcp-not-working-in-hadoop-2-6-0)
but the only advice I got was something I'd already tried that wasn't working for me.

It's a stock install (unpacked the hadoop archive, and set HADOOP_PREFIX) and is running Java
and Pig jobs just fine. For distcp, I get no filesystem for scheme. I set HADOOP_CLASSPATH
to /opt/hadoop/share/hadoop/tools/lib/* and now hadoop classpath shows that folder, but I
now get ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem

Could anyone give me some clues where to look? Since the error changed, I think the aws jar
is being included (otherwise what caused the filesystem to be registered), but when run on
the actual cluster it seems like it's not packaging the right jars.


View raw message