cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Chen <min.c...@citrix.com>
Subject Re: Object based Secondary storage.
Date Fri, 07 Jun 2013 04:02:40 GMT
John,
  We are not able to successfully download file that was uploaded to Riak CS with TransferManager
using S3cmd. Same error as we encountered using amazon s3 java client due to the incompatible
ETAG format ( - and _ difference).

Thanks
-min



On Jun 6, 2013, at 5:40 PM, "John Burwell" <jburwell@basho.com> wrote:

> Edison,
> 
> Riak CS and S3 seed their hashes differently -- causing the form to appear slightly different.
 In particular, Riak CS uses URI-safe base64 encoding which explains why the ETag values contain
"-"s instead of "_"s.  From a client perspective, the ETags are treated as opaque strings
that are passed through to the server for processing and compared strictly for equality. 
Therefore, the form of the hash will not cause the client to choke, and the Riak CS behavior
you are seeing is S3 API compatible (see http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html
for more details).  
> 
> Were you able to successfully download the file from Riak CS using s3cmd?
> 
> Thanks,
> -John
> 
> 
> On Jun 6, 2013, at 6:57 PM, Edison Su <Edison.su@citrix.com> wrote:
> 
>> The Etag created by both RIAK CS and Amazon S3 seems a little bit different, in case
of multi part upload.
>> 
>> Here is the result I tested on both RIAK CS and Amazon S3, with s3cmd.
>> Test environment:
>> S3cmd: version: version 1.5.0-alpha1
>> Riak cs:
>> Name        : riak
>> Arch        : x86_64
>> Version     : 1.3.1
>> Release     : 1.el6
>> Size        : 40 M
>> Repo        : installed
>> From repo   : basho-products
>> 
>> The command I used to put:
>> s3cmd put some-file s3://some-path --multipart-chunk-size-mb=100 -v -d
>> 
>> The etag created for the file, when using Riak CS is WxEUkiQzTWm_2C8A92fLQg==
>> 
>> EBUG: Sending request method_string='POST', uri='http://imagestore.s3.amazonaws.com/tmpl/1/1/routing-1/test?uploadId=kfDkh7Q_QCWN7r0ZTqNq4Q==',
headers={'content-length': '309', 'Authorization': 'AWS OYAZXCAFUC1DAFOXNJWI:xlkHI9tUfUV/N+Ekqpi7Jz/pbOI=',
'x-amz-date': 'Thu, 06 Jun 2013 22:54:28 +0000'}, body=(309 bytes)
>> DEBUG: Response: {'status': 200, 'headers': {'date': 'Thu, 06 Jun 2013 22:40:09 GMT',
'content-length': '326', 'content-type': 'application/xml', 'server': 'Riak CS'}, 'reason':
'OK', 'data': '<?xml version="1.0" encoding="UTF-8"?><CompleteMultipartUploadResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://imagestore.s3.amazonaws.com/tmpl/1/1/routing-1/test</Location><Bucket>imagestore</Bucket><Key>tmpl/1/1/routing-1/test</Key><ETag>kfDkh7Q_QCWN7r0ZTqNq4Q==</ETag></CompleteMultipartUploadResult>'}
>> 
>> While the etag created by Amazon S3 is: &quot;70e1860be687d43c039873adef4280f2-3&quot;
>> 
>> DEBUG: Sending request method_string='POST', uri='/fixes/icecake/systdfdfdfemvm.iso1?uploadId=vdkPSAtaA7g.fdfdfdfdf..iaKRNW_8QGz.bXdfdfdfdfdfkFXwUwLzRcG5obVvJFDvnhYUFdT6fYr1rig--',

>> DEBUG: Response: {'status': 200, 'headers': {, 'server': 'AmazonS3', 'transfer-encoding':
'chunked', 'connection': 'Keep-Alive', 'x-amz-request-id': '8DFF5D8025E58E99', 'cache-control':
'proxy-revalidate', 'date': 'Thu, 06 Jun 2013 22:39:47 GMT', 'content-type': 'application/xml'},
'reason': 'OK', 'data': '<?xml version="1.0" encoding="UTF-8"?>\n\n<CompleteMultipartUploadResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>http://fdfdfdfdfdfdf</Location>Key>fixes/icecake/systemvm.iso1</Key><ETag>&quot;70e1860be687d43c039873adef4280f2-3&quot;</ETag></CompleteMultipartUploadResult>'}
>> 
>> So the etag created on Amazon S3 has "-"(dash) in it, but there is only "_" (underscore)
on Riak cs. 
>> 
>> Do you know the reason? What should we need to do to make it compatible with Amazon
S3 SDK?
>> 
>>> -----Original Message-----
>>> From: John Burwell [mailto:jburwell@basho.com]
>>> Sent: Thursday, June 06, 2013 2:03 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: Object based Secondary storage.
>>> 
>>> Min,
>>> 
>>> Are you calculating the MD5 or letting the Amazon client do it?
>>> 
>>> Thanks,
>>> -John
>>> 
>>> On Jun 6, 2013, at 4:54 PM, Min Chen <min.chen@citrix.com> wrote:
>>> 
>>>> Thanks Tom. Indeed I have a S3 question that need some advise from
>>>> some S3 experts. To support upload object > 5G, I have used
>>>> TransferManager.upload to upload object to S3, upload went fine and
>>>> object are successfully put to S3. However, later on when I am using
>>>> "s3cmd get <object key>" to retrieve this object, I always got this
exception:
>>>> 
>>>> "MD5 signatures do not match: computed=Y, received="X"
>>>> 
>>>> It seems that Amazon S3 kept a different Md5 sum for the multi-part
>>>> uploaded object. We have been using Riak CS for our S3 testing. If I
>>>> changed to not using multi-part upload and directly invoking S3
>>>> putObject, I will not run into this issue. Do you have such experience
>>> before?
>>>> 
>>>> -min
>>>> 
>>>> On 6/6/13 1:56 AM, "Thomas O'Dowd" <tpodowd@cloudian.com> wrote:
>>>> 
>>>>> Thanks Min. I've printed out the material and am reading new threads.
>>>>> Can't comment much yet until I understand things a bit more.
>>>>> 
>>>>> Meanwhile, feel free to hit me up with any S3 questions you have. I'm
>>>>> looking forward to playing with the object_store branch and testing
>>>>> it out.
>>>>> 
>>>>> Tom.
>>>>> 
>>>>> On Wed, 2013-06-05 at 16:14 +0000, Min Chen wrote:
>>>>>> Welcome Tom. You can check out this FS
>>>>>> 
>>>>>> 
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu
>>>>>> p+Obj
>>>>>> ec
>>>>>> t+Store+Plugin+Framework for secondary storage architectural work
>>>>>> t+Store+Plugin+done
>>>>>> in
>>>>>> object_store branch.You may also check out the following recent
>>>>>> threads regarding 3 major technical questions raised by community
as
>>>>>> well as our answers and clarification.
>>>>>> 
>>>>>> http://mail-archives.apache.org/mod_mbox/cloudstack-
>>> dev/201306.mbox/
>>>>>> %3C77
>>>>>> B3
>>>>>> 
>>> 37AF224FD84CBF8401947098DD87036A76%40SJCPEX01CL01.citrite.net%3E
>>>>>> 
>>>>>> http://mail-archives.apache.org/mod_mbox/cloudstack-
>>> dev/201306.mbox/
>>>>>> %3CCD
>>>>>> D2
>>>>>> 2955.3DDDC%25min.chen%40citrix.com%3E
>>>>>> 
>>>>>> http://mail-archives.apache.org/mod_mbox/cloudstack-
>>> dev/201306.mbox/
>>>>>> %3CCD
>>>>>> D2
>>>>>> 300D.3DE0C%25min.chen%40citrix.com%3E
>>>>>> 
>>>>>> 
>>>>>> That branch is mainly worked on by Edison and me, and we are at PST
>>>>>> timezone.
>>>>>> 
>>>>>> Thanks
>>>>>> -min
>>>>> --
>>>>> Cloudian KK - http://www.cloudian.com/get-started.html
>>>>> Fancy 100TB of full featured S3 Storage?
>>>>> Checkout the Cloudian(r) Community Edition!
>>>>> 
>>>> 
>> 
> 


Mime
View raw message