cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Chen <min.c...@citrix.com>
Subject Re: ACS + Riak-CS, works?
Date Thu, 31 Oct 2013 17:50:44 GMT
Hi John,
I have a related question on this issue for you regarding RiakCS. In 4.2.0 release, backup
snapshot to S3 is done through your s3xen plugin, where S3 single part upload API is used.
If customer is using RiakCS as their S3 object store, can they still backup snapshot larger
than 5GB? Based on your comments, it seems that they should be able to?

Thanks
-min

From: John Burwell <jburwell@basho.com<mailto:jburwell@basho.com>>
Reply-To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
Date: Thursday, October 31, 2013 10:30 AM
To: Darren Shepherd <darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>>
Cc: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
Subject: Re: ACS + Riak-CS, works?

Darren,

Since S3/Riak CS can receive the contents of a file out of order and re-assemble upon completion
of transfer of all parts, the underlying driver uses the TransferManager which automatically
breaks files up into multi-part uploads to increase parallelism to decrease the time required
to upload to the object store.  Therefore, multi-part uploads are not employed solely to upload
files larger than 5 GB.

Have you tried to upload the file to Riak CS using s3cmd outside of CloudStack?

Thanks,
-John

On Oct 31, 2013, at 1:23 PM, Darren Shepherd <darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>>
wrote:

My template is less than 5gb and it still uses multipart.  I actually don't think this is
a CloudStack issue but more of an issue with Riak-CS.  I was just wondering if anybody has
specifically ran the setup or knows more about Riak-CS and can tell me whats up.  I've been
trying off and on for the last week to try to get this working with 4.2 but it just keeps
failing.  I think I'll just try ceph rados gateway if I can't get this working.

Darren


On Wed, Oct 30, 2013 at 11:17 PM, Sanjeev Neelarapu <sanjeev.neelarapu@citrix.com<mailto:sanjeev.neelarapu@citrix.com>>
wrote:
Hi,

Cloudstack with Riak-CS works fine. If we register template more than 5GB we use multipart
upload to push the files to Riak. But this does not work for snapshots and volumes. This will
be fixed in CS4.2.1 . But for template registration it should work fine.

Thanks,
Sanjeev

-----Original Message-----
From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com<mailto:darren.s.shepherd@gmail.com>]
Sent: Thursday, October 31, 2013 11:18 AM
To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>
Subject: ACS + Riak-CS, works?

Has anybody gotten CloudStack and Riak-CS to work together.  No matter what I try when it
goes to download and save the template, after its posted a series of multi part uploads it
ends up sending a 403.  The access log looks like

127.0.0.1 - - [31/Oct/2013:05:41:11 +0000] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=14
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:28 +0000] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=15
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:42 +0000] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=16
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=17
HTTP/1.1" 200 0 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"
127.0.0.1 - - [31/Oct/2013:05:41:51 +0000] "PUT
/buckets/bucket1/objects/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2/uploads/LmEFZncZTJqgTdSe2FP5bw==?partNumber=18
HTTP/1.1" 403 311 "" "aws-sdk-java/1.3.22 Linux/3.8.0-31-generic OpenJDK_64-Bit_Server_VM/20.0-b12"

And then in the error log I always get

2013-10-31 05:41:51.867 [error]
<0.5117.0>@riak_cs_wm_common:maybe_create_user:223 Retrieval of user record for s3 failed.
Reason: disconnected

I'm able to interact with Riak-CS with s3cmd just fine but its not using multipart. I think
I'm going to try to download a new s3cmd that does multipart and see if even that will work.

Darren



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message