cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edison Su <Edison...@citrix.com>
Subject RE: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?
Date Tue, 09 Jul 2013 20:56:49 GMT


> -----Original Message-----
> From: John Burwell [mailto:jburwell@basho.com]
> Sent: Tuesday, July 09, 2013 1:31 PM
> To: dev@cloudstack.apache.org
> Cc: 'Chip Childers'
> Subject: Re: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?
> 
> Edison,
> 
> TL;DR The shorter path is to re-implement/fix the Swift driver.
> 
> Multipart upload would need to implemented in Swift, not in CloudStack.
> Therefore, such a change would need to be accepted and released by the
> OpenStack project before the 4.2.0 release.  We would also be stranding any
> of our current users who cannot or will not upgrade their Swift instances.
> 
> Knowing what it took to implement it in Riak CS, multi-part upload was a lot
> of work to implement in a Dynamo-based system.  The S3 has the following
> three-phase process:
> 
> 	1. Initiate a Upload: Declare the number of parts and their size
> 	2. Submit each part per the definition in Step 1 (e.g. 50 parts = 50
> HTTP PUTs)
> 	3. Complete the multi-part upload: Declare that all parts have been
> uploaded which causes the object to become available
> 
> In contrast, Swift uses HTTP chunking to solve the same problem with one
> API call.  In addition to providing all of the reliability guarantees of the S3 API,
> an implementor of S3 multipart uploads will have to provide a way to
> translate the 3-phase model into the single call model used by Swift.

Seems swift client can support upload files more than 5GB: 
http://fedoraproject.org/wiki/QA:Testcase_Swift_Upload_Large_File

Ok, so we should just re-implement/fix the existing swift driver.

> 
> Thanks,
> -John
> 
> On Jul 9, 2013, at 4:15 PM, Edison Su <Edison.su@citrix.com> wrote:
> 
> > Could they(swiftstack) help us, or guide us on how to implement multi-part
> upload?
> >
> >> -----Original Message-----
> >> From: Sebastien Goasguen [mailto:runseb@gmail.com]
> >> Sent: Tuesday, July 09, 2013 1:07 PM
> >> To: dev@cloudstack.apache.org
> >> Cc: 'Chip Childers'
> >> Subject: Re: Swift in 4.2 is broken, anybody wants it to be supported in 4.2?
> >>
> >> if swift does not work anymore in 4.0 or 4.1 maybe be should inform
> >> swiftstack:
> >> http://swiftstack.com/cloudstack/
> >>
> >>
> >> On Jul 9, 2013, at 3:57 PM, John Burwell <jburwell@basho.com> wrote:
> >>
> >>> Edison,
> >>>
> >>> Swift does not support S3 multi-part uploads [1] which CloudStack
> >>> must use
> >> in order to store files larger than 5 GB.  Therefore, using the
> >> Swift's S3 compatibility layer is not a viable workaround.
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> [1]:  https://wiki.openstack.org/wiki/Swift/APIFeatureComparison
> >>>
> >>> On Jul 9, 2013, at 2:12 PM, Edison Su <Edison.su@citrix.com> wrote:
> >>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Chip Childers [mailto:chip.childers@sungard.com]
> >>>>> Sent: Monday, July 08, 2013 1:26 PM
> >>>>> To: Edison Su
> >>>>> Cc: <dev@cloudstack.apache.org>
> >>>>> Subject: Re: Swift in 4.2 is broken, anybody wants it to be
> >>>>> supported in
> >> 4.2?
> >>>>>
> >>>>> On Mon, Jul 08, 2013 at 05:15:19PM +0000, Edison Su wrote:
> >>>>>>
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Chip Childers [mailto:chip.childers@sungard.com]
> >>>>>>> Sent: Monday, July 08, 2013 6:46 AM
> >>>>>>> To: <dev@cloudstack.apache.org>; Edison Su
> >>>>>>> Subject: Re: Swift in 4.2 is broken, anybody wants it to
be
> >>>>>>> supported in
> >>>>> 4.2?
> >>>>>>>
> >>>>>>> On Mon, Jul 8, 2013 at 9:22 AM, David Nalley <david@gnsa.us>
> wrote:
> >>>>>>>> On Wed, Jul 3, 2013 at 5:29 PM, Edison Su
> >>>>>>>> <Edison.su@citrix.com>
> >>>>> wrote:
> >>>>>>>>> Due to object store refactor, Swift is broken. The
reason, is
> >>>>>>>>> that, we only
> >>>>>>> have S3 test environment in our lab, so only S3 is tested
for now.
> >>>>>>>>> Before adding the feature back, I'd better ask from,
the
> >>>>>>>>> community, do
> >>>>>>> we want to support Swift? If so, which version of Swift?
This
> >>>>>>> will take some efforts to support Swift, are there any
> >>>>>>> volunteers can help the
> >>>>> integration?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Whats the bug ID for this?
> >>>>>>>> Unplanned/Unannounced deprecation of a feature is a
blocker
> IMO.
> >>>>>>>> It engenders a bad relationship with our users, and
strands
> >>>>>>>> them on previous versions with no good migration/upgrade
path.
> >>>>>>>>
> >>>>>>>> --David
> >>>>>>>>
> >>>>>>>
> >>>>>>> Edison, How broken is it?  Is it shorter to fix or revert
the
> >>>>>>> object store changes?
> >>>>>> It's not working at all. Not sure, revert object store will
> >>>>>> change it or not, as
> >>>>> this feature is not tested by QA for a long time.
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> So any idea what the effort of fixing it looks like?  I mean, just
> >>>>> because it
> >>>>
> >>>> If it's ok to use S3 api talking to swift, then there is zero
> >>>> effort to support
> >> Swift.
> >>>> But who will make the decision?
> >>>>
> >>>>> wasn't tested in the last couple of releases doesn't necessarily
> >>>>> mean that it wasn't working.  As Sudha mentioned, it wasn't tested
> >>>>> only because of a lack of change that triggered the expected need
> >>>>> to perform regression testing of that feature.
> >>>>>
> >>>>> I believe that this was an honest mistake, but we need to figure
> >>>>> out what to do.  I'm -1 on us saying "we'll drop Swift support".
> >>>>> If necessary, I'd say that we need to roll back the object-store
> >>>>> branch merge...  I don't want to see that happen though.  That's
> >>>>> why I'm asking
> >> about effort to fix it.
> >>>>>
> >>>>> -chip
> >>>
> >


Mime
View raw message