Return-Path: 
 <common-dev-return-92864-apmail-hadoop-common-dev-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-dev-archive@www.apache.org
Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BC86119FEC
	for <apmail-hadoop-common-dev-archive@www.apache.org>;
 Mon, 25 Apr 2016 21:12:15 +0000 (UTC)
Received: (qmail 85640 invoked by uid 500); 25 Apr 2016 21:12:13 -0000
Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org
Received: (qmail 85494 invoked by uid 500); 25 Apr 2016 21:12:13 -0000
Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-dev-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-dev-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-dev@hadoop.apache.org>
List-Id: <common-dev.hadoop.apache.org>
Reply-To: common-dev@hadoop.apache.org
Delivered-To: mailing list common-dev@hadoop.apache.org
Received: (qmail 85426 invoked by uid 99); 25 Apr 2016 21:12:13 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Apr 2016 21:12:13 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id CE5CE2C1F54
	for <common-dev@hadoop.apache.org>; Mon, 25 Apr 2016 21:12:12 +0000 (UTC)
Date: Mon, 25 Apr 2016 21:12:12 +0000 (UTC)
From: "Jenny Dong (JIRA)" <jira@apache.org>
To: common-dev@hadoop.apache.org
Message-ID: <JIRA.12962394.1461618681000.28044.1461618732842@Atlassian.JIRA>
In-Reply-To: <JIRA.12962394.1461618681000@Atlassian.JIRA>
References: <JIRA.12962394.1461618681000@Atlassian.JIRA>
 <JIRA.12962394.1461618681655@arcas>
Subject: [jira] [Created] (HADOOP-13060) While trying to perform a Distcp
 command, we see the error Exception in thread "main"
 java.lang.NoSuchMethodError:
 com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Jenny Dong created HADOOP-13060:
-----------------------------------

             Summary: While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
                 Key: HADOOP-13060
                 URL: https://issues.apache.org/jira/browse/HADOOP-13060
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 2.7.1
            Reporter: Jenny Dong


While trying to do a distcp from our native HDFS cluster to S3, we get the following error/stacktrace : 

We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors seen in HADOOP-12420 + other authentication errors).

Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
	at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
	at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
	at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)

I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem created a ThreadPoolExecutor (which implements AbstractExecutorService which implements ExecutorService). I also checked on the classpath to make sure the version of the jars being picked up is what I expected. 

Help would be much appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)