Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: domain of jburwell@basho.com designates
 209.85.216.175 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\))
Subject: Re: [MERGE]object_store branch into master
From: John Burwell <jburwell@basho.com>
In-Reply-To: <77B337AF224FD84CBF8401947098DD870235F1@SJCPEX01CL01.citrite.net>
Date: Tue, 21 May 2013 11:51:15 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <C637DD8A-9929-474E-94C9-EEA043649AD8@basho.com>
References: <77B337AF224FD84CBF8401947098DD87021BD5@SJCPEX01CL01.citrite.net>
 <CAKprHVbM0ePzv-HsC3WdR_CAz9GYW1ZKjvGrX9_OS=-FjC6AdA@mail.gmail.com>
 <20130520175238.GQ56667@USLT-205755.sungardas.corp>
 <195439EE-15D8-447C-ADFA-A548CEE9E24F@basho.com>
 <77B337AF224FD84CBF8401947098DD87023426@SJCPEX01CL01.citrite.net>
 <77B337AF224FD84CBF8401947098DD870235F1@SJCPEX01CL01.citrite.net>
To: dev@cloudstack.apache.org

Edison,

Thanks, I will start going through it today.  Based on other $dayjob =
responsibilities, it may take me a couple of days.

Thanks,
-John

On May 20, 2013, at 6:15 PM, Edison Su <Edison.su@citrix.com> wrote:

>=20
>=20
>> -----Original Message-----
>> From: Edison Su [mailto:Edison.su@citrix.com]
>> Sent: Monday, May 20, 2013 2:30 PM
>> To: dev@cloudstack.apache.org
>> Subject: RE: [MERGE]object_store branch into master
>>=20
>>=20
>>=20
>>> -----Original Message-----
>>> From: John Burwell [mailto:jburwell@basho.com]
>>> Sent: Monday, May 20, 2013 12:56 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [MERGE]object_store branch into master
>>>=20
>>> All,
>>>=20
>>> Since this change is so large, it makes reviewing and commenting in
>>> detail extremely difficult.  Would it be possible to push this patch
>>> through Review Board to ease comprehension and promote a =
conversation
>> about this patch?
>>=20
>> We can try to push it into Review Board.
>=20
> The review board url is: https://reviews.apache.org/r/11277/, 25 =
pages...
>=20
>>=20
>>>=20
>>> Reading through the FS, I have the following questions regarding the
>>> operation of the NFS cache:
>>>=20
>>> What happens if/when the disk space of the NFS cache is exhausted?
>>> What are the sizing recommendations/guidelines for it?
>>> What strategy is used to age files out of the NFS cache?
>> As usual, admin can have multiple NFS secondary storages, admin can =
also
>> add multiple NFS cache storages. The NFS cache storage capacity plan =
will be
>> the same as NFS secondary storage.
>> If there multiple NFS cache storages, the current strategy will =
randomly
>> choose one of them. Currently, no clean up/aging out strategy =
implemented
>> yet But the situation can be improved: most of cached object can be =
deleted
>> after accessed once. Take template as example, if zone wide storage =
is used,
>> put template on cache storage has little value, as once the template =
is
>> downloaded into primary storage, suddenly all the hypervisor host can =
access
>> it.
>> I think the simple LRU algorithm to delete cached objects should be =
enough.
>> It can be added later, the cache storage has its own pom project, =
it's place to
>> add more intelligence.
>>=20
>>> If two processes, process1 and process2, are both using a template,
>>> templateA, will both processes reference the same file in the NFS
>>> cache?  If
>> It's possible, that one template can be downloaded into cache storage =
twice,
>> in case of concurrent accessed by two processes. The current
>> implementation is that, if two processes want to download the same
>> template from s3 into one primary storage at the same time, there is =
only
>> one template will be downloaded into cache storage. While, if two =
processes
>> want to download the same template into different primary storage, =
the
>> template will be cached twice.
>>> they are reading from the same file and process1 finishes before
>>> process2, will process1 attempt to delete process2?
>>=20
>> There is no way to delete while read, as each cached object has its =
own state
>> machine. If it's accessed by one process, the state will be changed =
to
>> "Copying", you can't delete an object when it's in "Copying" state.
>>=20
>>> If a file transfer from the NFS cache to the object store fails, =
what
>>> is the recovery/retry strategy?  What durability guarantees will
>>> CloudStack supply when a snapshot, template, or ISO is in the cache,
>>> but can't be written to the object store?
>>=20
>> The error handling of cache storage shouldn't be different than =
without
>> cache storage. For example, directly backup snapshot from primary =
storage
>> to S3, without cache storage. If backup failed, then the whole =
process failed,
>> user needs to do it again through cloudstack API. So in cache storage =
case, if
>> push object from cache storage to s3 failed, then the whole backup =
process
>> failed.
>>=20
>>> What will be the migration strategy for the objects contained in S3
>>> buckets/Swift containers from pre-4.2.0 instances?  Currently,
>>> CloudStack tracks a mapping between these objects and templates/ISOs
>>> in the template_switt_ref and template_s3_ref table.
>>=20
>> We need to migrate DB from existing template_s3_ref to
>> template_store_ref, and put all the s3 information into image_store =
and
>> image_store_details tables.
>>=20
>>>=20
>>> Finally, does the S3 implementation use multi-part upload to =
transfer
>>> files to the object store?  If not, the implementation will be =
limited
>>> to storing files no larger than 5GB in size.
>> Oh, this is something we don't know yet. We haven't try to upload a
>> template which is large than 5GB, so haven't met this issue.
>> Could you help to hack it up?:)
>>=20
>>>=20
>>> Thanks,
>>> -John
>>>=20
>>> On May 20, 2013, at 1:52 PM, Chip Childers =
<chip.childers@sungard.com>
>>> wrote:
>>>=20
>>>> On Fri, May 17, 2013 at 08:19:57AM -0400, David Nalley wrote:
>>>>> On Fri, May 17, 2013 at 4:11 AM, Edison Su <Edison.su@citrix.com>
>> wrote:
>>>>>> Hi all,
>>>>>>    Min and I worked on object_store branch during the last one
>>>>>> and half
>>> month. We made a lot of refactor on the storage code, mostly related
>>> to secondary storage, but also on the general storage framework. The
>>> following goals are made:
>>>>>>=20
>>>>>> 1.       An unified storage framework. Both secondary
>>> storages(nfs/s3/swift etc) and primary storages will share the same
>>> plugin model, the same interface. Add any other new storages into
>>> cloudstack will much easier and straightforward.
>>>>>>=20
>>>>>> 2.       The storage interface  between mgt server and resource =
is
>> unified,
>>> currently there are only 5 commands send out by mgt server:
>>>=20
>> copycommand/createobjectcommand/deletecommand/attachcommand/de
>>> ttachcommand, and each storage vendor can decode/encode all the
>>> entities(volume/snapshot/storage pool/ template etc) by its own.
>>>>>>=20
>>>>>> 3.       NFS secondary storage is not explicitly depended on by =
other
>>> components. For example, when registering template into S3, template
>>> will be write into S3 directly, instead of storing into nfs =
secondary
>>> storage, then push to S3. If s3 is used as secondary storage, then =
nfs
>>> storage will be used as cache storage, but from other components =
point
>>> of view, cache storage is invisible. So, it's possible to make nfs
>>> storage as optional if s3 is used for certain hypervisors.
>>>>>> The detailed FS is at
>>>>>>=20
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+Backu
>>>>>> p+Object+Store+Plugin+Framework
>>>>>> The test we did:
>>>>>>=20
>>>>>> 1.       We modified marvin to use new storage api
>>>>>>=20
>>>>>> 2.       Test_volume and test_vm_life_cycle, test_template under
>> smoke
>>> test folder are executed against xenserver/kvm/vmware and devcloud,
>>> some of them are failed, it's partly due to bugs introduced by our
>>> code, partly master branch itself has issue(e.g. resizevolume =
doesn't
>>> work). We want to fix these issues after merging into master.
>>>>>>=20
>>>>>> The basic follow does work: create user vm, attach/detach volume,
>>> register template, create template from volume/snapshot, take
>>> snapshot, create volume from snapshot.
>>>>>> It's a huge change, around 60k LOC patch, to review the code, you
>>>>>> can
>>> try: git diff master..object_store, will show all the diff.
>>>>>> Comments/feedback are welcome. Thanks.
>>>>>>=20
>>>>>>=20
>>>>>=20
>>>>>=20
>>>>> Given the amount of change, can we get at least a BVT run against
>>>>> your branch done before merge?
>>>>>=20
>>>>> --David
>>>>>=20
>>>>=20
>>>> +1 to BVT please.
>=20