hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Reid (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-13401) Snapshot export to S3 fails with jets3t error - The Content-MD5 you specified did not match what we received.
Date Fri, 03 Apr 2015 19:05:53 GMT

     [ https://issues.apache.org/jira/browse/HBASE-13401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joseph Reid updated HBASE-13401:
--------------------------------
    Description: 
We're running into issues exporting snapshots of large tables to Amazon S3.

The snapshot completes successfully, but the snapshot export job throws jets3t errors when
we attempt to export to S3.  

Error snippet, from job log:
{code}
2015-04-03 16:59:16,425 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000008_1,
Status : FAILED
Error: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 Error
Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5
you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
  at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
  at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
  at org.apache.hadoop.fs.s3native.$Proxy19.storeFile(Unknown Source)
  at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
  at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:200)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:140)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:89)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400,
ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The
Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
  at org.jets3t.service.S3Service.putObject(S3Service.java:2267)
  at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
  ... 21 more

2015-04-03 17:03:50,613 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000010_1,
Status : FAILED
AttemptID:attempt_1426532296228_55454_m_000010_1 Timed out after 300 secs
{\code}

We've verified that exports to other clusters from these same snapshots work fine.  Thus the
issue appears to lie within the snapshot export utility, jets3t, and S3.

"The Content-MD5 you specified did not match what we received" seems to indicate that the
snapshot changed between when the upload started and the error.   Is that possible?  Why would
a snapshot change?  

Our jets3t.properties:
{code}
storage-service.internal-error-retry-max=5
storage-service.disable-live-md5=false
threaded-service.max-thread-count=20
threaded-service.admin-max-thread-count=20
s3service.max-thread-count=20
s3service.admin-max-thread-count=20
{\code}

Related to:
[Discussion on jets3t user group|https://groups.google.com/forum/#!topic/jets3t-users/Bg2qh7OdE2U]

  was:
We're running into issues exporting snapshots of large tables to Amazon S3.

The snapshot completes successfully, but the snapshot export job throws jets3t errors when
we attempt to export to S3.  

Error snippet, from job log:
{code}
2015-04-03 16:59:16,425 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000008_1,
Status : FAILED
Error: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 Error
Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The Content-MD5
you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
  at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
  at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
  at org.apache.hadoop.fs.s3native.$Proxy19.storeFile(Unknown Source)
  at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
  at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:200)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:140)
  at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:89)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400,
ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The
Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
  at org.jets3t.service.S3Service.putObject(S3Service.java:2267)
  at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
  ... 21 more

2015-04-03 17:03:50,613 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000010_1,
Status : FAILED
AttemptID:attempt_1426532296228_55454_m_000010_1 Timed out after 300 secs
{\code}

We've verified that exports to other clusters from these same snapshots work fine.  Thus the
issue appears to lie within the snapshot export utility, jets3t, and S3.

"The Content-MD5 you specified did not match what we received" seems to indicate that the
snapshot changed between when the upload started and the error.   Can that be?  

Related to:
[Discussion on jets3t user group|https://groups.google.com/forum/#!topic/jets3t-users/Bg2qh7OdE2U]


> Snapshot export to S3 fails with jets3t error - The Content-MD5 you specified did not
match what we received.
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-13401
>                 URL: https://issues.apache.org/jira/browse/HBASE-13401
>             Project: HBase
>          Issue Type: Bug
>          Components: hbase, mapreduce
>    Affects Versions: 0.98.0
>         Environment: CentOS 6.5, Hadoop 2.4.0, Yarn + MapReduce2 2.4.0, Hortonworks Data
Platform 2.1.2
>            Reporter: Joseph Reid
>
> We're running into issues exporting snapshots of large tables to Amazon S3.
> The snapshot completes successfully, but the snapshot export job throws jets3t errors
when we attempt to export to S3.  
> Error snippet, from job log:
> {code}
> 2015-04-03 16:59:16,425 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000008_1,
Status : FAILED
> Error: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3
Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml
version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The
Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
>   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleS3ServiceException(Jets3tNativeFileSystemStore.java:405)
>   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:115)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at org.apache.hadoop.fs.s3native.$Proxy19.storeFile(Unknown Source)
>   at org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:221)
>   at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
>   at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:103)
>   at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:200)
>   at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:140)
>   at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:89)
>   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode:
400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>BadDigest</Code><Message>The
Content-MD5 you specified did not match what we received.</Message><ExpectedDigest>CWiSsgzVAJyzPy2oT8u4Ag==</ExpectedDigest><CalculatedDigest>2DIsv6jZJ8FuGtalOO8SPA==</CalculatedDigest><RequestId>CA325C738970C313</RequestId><HostId>tnE+O1zPZovaQWMhCuM4lkX0h/wN9173FQ7omxZzLb6eH0OCHASyan+mb8WBJkNn</HostId></Error>
>   at org.jets3t.service.S3Service.putObject(S3Service.java:2267)
>   at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:113)
>   ... 21 more
> 2015-04-03 17:03:50,613 INFO  [main] mapreduce.Job: Task Id : attempt_1426532296228_55454_m_000010_1,
Status : FAILED
> AttemptID:attempt_1426532296228_55454_m_000010_1 Timed out after 300 secs
> {\code}
> We've verified that exports to other clusters from these same snapshots work fine.  Thus
the issue appears to lie within the snapshot export utility, jets3t, and S3.
> "The Content-MD5 you specified did not match what we received" seems to indicate that
the snapshot changed between when the upload started and the error.   Is that possible?  Why
would a snapshot change?  
> Our jets3t.properties:
> {code}
> storage-service.internal-error-retry-max=5
> storage-service.disable-live-md5=false
> threaded-service.max-thread-count=20
> threaded-service.admin-max-thread-count=20
> s3service.max-thread-count=20
> s3service.admin-max-thread-count=20
> {\code}
> Related to:
> [Discussion on jets3t user group|https://groups.google.com/forum/#!topic/jets3t-users/Bg2qh7OdE2U]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message