cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Burwell <jburw...@basho.com>
Subject Re: ACS + Riak-CS, works?
Date Thu, 31 Oct 2013 17:58:05 GMT
Min,

I apologize for the confusions.  Files greater than 5 GB must uploaded using multipart and
no part can exceed 5 GB in size.  The original S3 implementation in 4.1.0 didn't support multi-part
uploads for templates.  Therefore, I didn't implement it in the s3xen plugin.  Now that we
support it for all other operations, the s3xen plugin needs to be retrofitted for multipart.

Thanks,
-John
  
On Oct 31, 2013, at 1:50 PM, Min Chen <min.chen@citrix.com> wrote:

> Hi John,
> I have a related question on this issue for you regarding RiakCS. In 4.2.0 release, backup
snapshot to S3 is done through your s3xen plugin, where S3 single part upload API is used.
If customer is using RiakCS as their S3 object store, can they still backup snapshot larger
than 5GB? Based on your comments, it seems that they should be able to?
> 
> Thanks
> -min
> 
> From: John Burwell <jburwell@basho.com<mailto:jburwell@basho.com>>
> Reply-To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> Date: Thursday, October 31, 2013 10:30 AM
> To: Darren Shepherd <darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>>
> Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
> Subject: Re: ACS + Riak-CS, works?
> 
> Darren,
> 
> Since S3/Riak CS can receive the contents of a file out of order and re-assemble upon
completion of transfer of all parts, the underlying driver uses the TransferManager which
automatically breaks files up into multi-part uploads to increase parallelism to decrease
the time required to upload to the object store.  Therefore, multi-part uploads are not employed
solely to upload files larger than 5 GB.
> 
> Have you tried to upload the file to Riak CS using s3cmd outside of CloudStack?
> 
> Thanks,
> -John
> 
> On Oct 31, 2013, at 1:23 PM, Darren Shepherd <darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>>
wrote:
> 
> My template is less than 5gb and it still uses multipart.  I actually don't think this
is a CloudStack issue but more of an issue with Riak-CS.  I was just wondering if anybody
has specifically ran the setup or knows more about Riak-CS and can tell me whats up.  I've
been trying off and on for the last week to try to get this working with 4.2 but it just keeps
failing.  I think I'll just try ceph rados gateway if I can't get this working.
> 
> Darren
> 
> 
> On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <sanjeev.neelarapu@citrix.com<mailto:sanjeev.neelarapu@citrix.com>>
wrote:
> Hi,
> 
> Cloudstack with Riak-CS works fine. If we register template more than 5GB we use multipart
upload to push the files to Riak. But this does not work for snapshots and volumes. This will
be fixed in CS4.2.1 . But for template registration it should work fine.
> 
> Thanks,
> Sanjeev
> 
> -----Original Message-----
> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>]
> Sent: Thursday, October 31, 2013 11:18 AM
> To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>
> Subject: ACS + Riak-CS, works?
> 
> Has anybody gotten CloudStack and Riak-CS to work together.  No matter what I try when
it goes to download and save the template, after its posted a series of multi part uploads
it ends up sending a 403.  The access log looks like
> 
> 127.0.0.1 - - [31/Oct/2013:05:41:11 +0000] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:28 +0000] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:42 +0000] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
> HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
> 127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT
> /buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
> HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
> 
> And then in the error log I always get
> 
> 2013-10-31 05:41:51.867 [error]
> <0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for
s3 failed. Reason: disconnected
> 
> I'm able to interact with Riak-CS with s3cmd just fine but its not using multipart. I
think I'm going to try to download a new s3cmd that does multipart and see if even that will
work.
> 
> Darren
> 
> 


Mime
View raw message