Return-Path: 
 <issues-return-190002-apmail-hbase-issues-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-issues-archive@www.apache.org
Delivered-To: apmail-hbase-issues-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDEEA17480
	for <apmail-hbase-issues-archive@www.apache.org>;
 Fri,  3 Apr 2015 18:33:58 +0000 (UTC)
Received: (qmail 91041 invoked by uid 500); 3 Apr 2015 18:33:53 -0000
Delivered-To: apmail-hbase-issues-archive@hbase.apache.org
Received: (qmail 90991 invoked by uid 500); 3 Apr 2015 18:33:53 -0000
Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@hbase.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@hbase.apache.org>
List-Post: <mailto:issues@hbase.apache.org>
List-Id: <issues.hbase.apache.org>
Delivered-To: mailing list issues@hbase.apache.org
Received: (qmail 90979 invoked by uid 99); 3 Apr 2015 18:33:53 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2015 18:33:53 +0000
Date: Fri, 3 Apr 2015 18:33:53 +0000 (UTC)
From: "Andrew Purtell (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12788052.1428082848000.120060.1428086033669@Atlassian.JIRA>
In-Reply-To: <JIRA.12788052.1428082848000@Atlassian.JIRA>
References: <JIRA.12788052.1428082848000@Atlassian.JIRA>
 <JIRA.12788052.1428082848044@arcas>
Subject: [jira] [Commented] (HBASE-13400) HBase Snapshot export to S3 fails
 with Content-MD5 errors.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HBASE-13400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14394859#comment-14394859 ] 

Andrew Purtell commented on HBASE-13400:
----------------------------------------

And, although I do not know this for sure regarding this particular version of HDP, since I don't know the details of HDP, but vendors have been known to change the Hadoop version used to build HBase from the default. (smile) So even if we did accept responsibility for the S3 filesystem, even though most of us cannot commit changes to fix it, since it's not HBase code, we would also be accepting responsibility for a vendor frob of dependency version changes.


> HBase Snapshot export to S3 fails with Content-MD5 errors.
> ----------------------------------------------------------
>
>                 Key: HBASE-13400
>                 URL: https://issues.apache.org/jira/browse/HBASE-13400
>             Project: HBase
>          Issue Type: Bug
>          Components: Filesystem Integration, hadoop2
>    Affects Versions: 0.98.0
>         Environment: CentOS 6.5, Hortonworks Data Platform 2.1.2, Hadoop 2.4.0
>            Reporter: Joseph Reid
>
> We're running into issues exporting snapshots of large tables to Amazon S3.
> The snapshot completes successfully, but the snapshot export job runs into errors with jets3t when we attempt to export to S3.  
> Error snippet, from job log:
> {code}
> 2015-04-03 16:59:16,425 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000008_1, Status : FAILED
> Error: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
> 	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
> 	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> 	at org.apache.hadoop.fs.s3native.$Proxy19.storeFile(Unknown Source)
> 	at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
> 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> 	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
> 	at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:200)
> 	at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:140)
> 	at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:89)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
> 	at org.jets3t.service.S3Service.putObject(S3Service.java:2267)
> 	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
> 	... 21 more
> 2015-04-03 17:03:50,613 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000010_1, Status : FAILED
> AttemptID:attempt_1426532296228_55454_m_000010_1 Timed out after 300 secs
> {\code}
> We've verified that exports to other clusters from these same snapshots work fine.  Thus the issue appears to lie within the snapshot export utility, jets3t, and S3.
> "The Content-MD5 you specified did not match what we received" seems to indicate that the snapshot changed between when the upload started and the error.   Can that be?  
> Related to:
> [Discussion on jets3t user group,.|https://groups.google.com/forum/#!topic/jets3t-users/Bg2qh7OdE2U]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)