Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Content-Type: multipart/alternative;
 boundary="===============5121467464449335840=="
MIME-Version: 1.0
Subject: Re: Review Request: S3-backed Secondary Storage
From: "John Burwell" <jburwell@basho.com>
To: "edison su" <edison.su@citrix.com>
Cc: "Prasanna Santhanam" <Prasanna.Santhanam@citrix.com>,
 "cloudstack" <cloudstack-dev@incubator.apache.org>,
 "Rohit Yadav" <rohit.yadav@citrix.com>, "David Nalley" <david@gnsa.us>,
 "John Burwell" <jburwell@basho.com>
Date: Fri, 14 Dec 2012 23:48:24 -0000
Message-ID: <20121214234824.7729.97267@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "John Burwell" <noreply@reviews.apache.org>
References: <20121214214913.7729.4032@reviews.apache.org>
In-Reply-To: <20121214214913.7729.4032@reviews.apache.org>
Reply-To: "John Burwell" <jburwell@basho.com>

--===============5121467464449335840==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8123/
-----------------------------------------------------------

(Updated Dec. 14, 2012, 11:48 p.m.)


Review request for cloudstack and edison su.


Description
-------

Backs NFS-based secondary storage with an S3-compatible object store. Perio=
dically, a reaper thread synchronizes templates and ISOs stored on a NFS se=
condary storage mount with a configured S3 object store. It also pushes sna=
pshots to the object store when they are created and downloads them in othe=
r zones on-demand. In addition to permitting the use of commodity or IaaS s=
torage solutions for static assets, it provides a means of automatically sy=
nchronizing template and ISO assets across multiple zones.

For more information about the design of the patch, please see the design d=
ocument (https://cwiki.apache.org/confluence/display/CLOUDSTACK/S3-backed+S=
econdary+Storage).


This addresses bug CLOUDSTACK-509.


Diffs
-----

  api/src/com/cloud/api/commands/ListS3sCmd.java 507053d =

  build/package.xml 09ed939 =

  pom.xml e5aa024 =

  server/src/com/cloud/configuration/ConfigurationManagerImpl.java 2d7dfe2 =

  server/src/com/cloud/server/ManagementServerImpl.java 117be57 =

  server/src/com/cloud/storage/dao/VMTemplateDao.java 1284ba1 =

  server/src/com/cloud/storage/dao/VMTemplateDaoImpl.java 5c71f1b =

  server/src/com/cloud/storage/s3/S3ManagerImpl.java 6b07254 =

  tools/marvin/marvin/cloudstackConnection.py 8c4e325 =

  tools/marvin/marvin/deployDataCenter.py 01235fe =

  utils/src/com/cloud/utils/db/DbUtil.java feef7b3 =


Diff: https://reviews.apache.org/r/8123/diff/


Testing
-------

I am submitting patch to begin the feedback process while we complete integ=
ration testing.  I have verified that it does not interfere with normal Clo=
udStack operations when S3-backed Secondary Storage is disabled (the defaul=
t setting) .  I have successfully tested operation of single zone template =
and ISO scenarios on devcloud described in the design document.  I am curre=
ntly working through some issues in our multi-zone test environment to comp=
lete all scenarios described.  The following are the known deficiencies of =
the current implementation which I plan to correct in a subsequent patch:

   * Cross zone garbage collection: When a global asset is deleted from one=
 zone's secondary storage, it is not deleted from the secondary storage of =
other zones which have downloaded it from the object store
   * S3 Configuration Update: The API only supports adding an object store =
configuration.  Users should be able to edit the access key, secret key, co=
nnection timeout. max error retries, and socket timeout.
   * Multi-threaded Uploads: Permit the upload of multiple assets to the ob=
ject store simultaneously to decrease the propagation latency across all zo=
nes.


Screenshots
-----------

S3 Configuration Form
  https://reviews.apache.org/r/8123/s/13/
S3 Enable Menu on the Zone Tab
  https://reviews.apache.org/r/8123/s/14/


Thanks,

John Burwell


--===============5121467464449335840==--