hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Armstrong <stephen.armstr...@linqia.com>
Subject distcp fails with s3n or s3a in 2.6.0
Date Fri, 08 May 2015 22:33:15 GMT

I've been stuck on a classpath problem for a while now, and am hoping for
some help. Hadoop 2.6.0 no longer seems to support distcp to S3 for me. I
posted this on Stack Overflow (
but the only advice I got was something I'd already tried that wasn't
working for me.

It's a stock install (unpacked the hadoop archive, and set HADOOP_PREFIX)
and is running Java and Pig jobs just fine. For distcp, I get no filesystem
for scheme. I set HADOOP_CLASSPATH to /opt/hadoop/share/hadoop/tools/lib/*
and now hadoop classpath shows that folder, but I now get
ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem

Could anyone give me some clues where to look? Since the error changed, I
think the aws jar is being included (otherwise what caused the filesystem
to be registered), but when run on the actual cluster it seems like it's
not packaging the right jars.


View raw message