cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Chen <min.c...@citrix.com>
Subject Re: ACS + Riak-CS, works?
Date Thu, 31 Oct 2013 18:08:16 GMT
Thanks for your clarification, John. Yes, we have fixed in 4.2.1.

-min

On 10/31/13 10:58 AM, "John Burwell" <jburwell@basho.com> wrote:

>Min,
>
>I apologize for the confusions.  Files greater than 5 GB must uploaded
>using multipart and no part can exceed 5 GB in size.  The original S3
>implementation in 4.1.0 didn't support multi-part uploads for templates.
>Therefore, I didn't implement it in the s3xen plugin.  Now that we
>support it for all other operations, the s3xen plugin needs to be
>retrofitted for multipart.
>
>Thanks,
>-John
>  
>On Oct 31, 2013, at 1:50 PM, Min Chen <min.chen@citrix.com> wrote:
>
>> Hi John,
>> I have a related question on this issue for you regarding RiakCS. In
>>4.2.0 release, backup snapshot to S3 is done through your s3xen plugin,
>>where S3 single part upload API is used. If customer is using RiakCS as
>>their S3 object store, can they still backup snapshot larger than 5GB?
>>Based on your comments, it seems that they should be able to?
>> 
>> Thanks
>> -min
>> 
>> From: John Burwell <jburwell@basho.com<mailto:jburwell@basho.com>>
>> Reply-To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>> Date: Thursday, October 31, 2013 10:30 AM
>> To: Darren Shepherd
>><darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>>
>> Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
>><dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>> Subject: Re: ACS + Riak-CS, works?
>> 
>> Darren,
>> 
>> Since S3/Riak CS can receive the contents of a file out of order and
>>re-assemble upon completion of transfer of all parts, the underlying
>>driver uses the TransferManager which automatically breaks files up into
>>multi-part uploads to increase parallelism to decrease the time required
>>to upload to the object store.  Therefore, multi-part uploads are not
>>employed solely to upload files larger than 5 GB.
>> 
>> Have you tried to upload the file to Riak CS using s3cmd outside of
>>CloudStack?
>> 
>> Thanks,
>> -John
>> 
>> On Oct 31, 2013, at 1:23 PM, Darren Shepherd
>><darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>> wrote:
>> 
>> My template is less than 5gb and it still uses multipart.  I actually
>>don't think this is a CloudStack issue but more of an issue with
>>Riak-CS.  I was just wondering if anybody has specifically ran the setup
>>or knows more about Riak-CS and can tell me whats up.  I've been trying
>>off and on for the last week to try to get this working with 4.2 but it
>>just keeps failing.  I think I'll just try ceph rados gateway if I can't
>>get this working.
>> 
>> Darren
>> 
>> 
>> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu
>><sanjeev.neelarapu@citrix.com<mailto:sanjeev.neelarapu@citrix.com>>
>>wrote:
>> Hi,
>> 
>> Cloudstack with Riak-CS works fine. If we register template more than
>>5GB we use multipart upload to push the files to Riak. But this does not
>>work for snapshots and volumes. This will be fixed in CS4.2.1 . But for
>>template registration it should work fine.
>> 
>> Thanks,
>> Sanjeev
>> 
>> -----Original Message-----
>> From: Darren Shepherd
>>[mailto:darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>]
>> Sent: Thursday, October 31, 2013 11:18 AM
>> To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>
>> Subject: ACS + Riak-CS, works?
>> 
>> Has anybody gotten CloudStack and Riak-CS to work together.  No matter
>>what I try when it goes to download and save the template, after its
>>posted a series of multi part uploads it ends up sending a 403.  The
>>access log looks like
>> 
>> 127.0.0.1 - - [31/Oct/2013:05:41:11 +0000] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=14
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:28 +0000] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=15
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:42 +0000] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=16
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=17
>> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT
>> 
>>/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2
>>209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw
>>==?partNumber=18
>> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic
>>OpenJDK_64-Bit_Server_VM/20.0-b12"
>> 
>> And then in the error log I always get
>> 
>> 2013-10-31 05:41:51.867 [error]
>> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user
>>record for s3 failed. Reason: disconnected
>> 
>> I'm able to interact with Riak-CS with s3cmd just fine but its not
>>using multipart. I think I'm going to try to download a new s3cmd that
>>does multipart and see if even that will work.
>> 
>> Darren
>> 
>> 
>


Mime
View raw message