incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Burwell" <>
Subject Re: Review Request: S3-backed Secondary Storage
Date Mon, 19 Nov 2012 06:37:54 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Nov. 19, 2012, 6:37 a.m.)

Review request for cloudstack.


Backs NFS-based secondary storage with an S3-compatible object store. Periodically, a reaper
thread synchronizes templates and ISOs stored on a NFS secondary storage mount with a configured
S3 object store. It also pushes snapshots to the object store when they are created and downloads
them in other zones on-demand. In addition to permitting the use of commodity or IaaS storage
solutions for static assets, it provides a means of automatically synchronizing template and
ISO assets across multiple zones.

For more information about the design of the patch, please see the design document (

This addresses bug CLOUDSTACK-509.


  .gitignore 9daca15 
  api/src/com/cloud/agent/api/ 9476d7d 
  api/src/com/cloud/agent/api/ 3fa8c2b 
  api/src/com/cloud/agent/api/ PRE-CREATION 
  api/src/com/cloud/agent/api/ PRE-CREATION 
  api/src/com/cloud/agent/api/ PRE-CREATION

  api/src/com/cloud/agent/api/ PRE-CREATION

  api/src/com/cloud/agent/api/to/ PRE-CREATION 
  api/src/com/cloud/api/ 78a3ded 
  api/src/com/cloud/api/ 4e8fbd8 
  api/src/com/cloud/api/commands/ PRE-CREATION 
  api/src/com/cloud/api/commands/ PRE-CREATION 
  api/src/com/cloud/api/response/ PRE-CREATION 
  api/src/com/cloud/resource/ 1065453 
  api/src/com/cloud/storage/ PRE-CREATION 
  build/package.xml 09ed939 
  client/WEB-INF/classes/resources/ 626e44a 
  client/tomcatconf/ 149547e 
  console-proxy/scripts/ e408378 
  core/pom.xml 15f0f7b 
  core/src/com/cloud/agent/resource/virtualnetwork/ 2dc7950 
  core/src/com/cloud/storage/ PRE-CREATION 
  core/src/com/cloud/storage/ 08dfafa 
  core/src/com/cloud/storage/ PRE-CREATION 
  core/src/com/cloud/storage/resource/ 155210d 
  patches/systemvm/debian/config/etc/sysctl.conf 7f945b0 
  plugins/hypervisors/xen/src/com/cloud/hypervisor/xen/resource/ d2db85c

  pom.xml 08f61f5 
  scripts/vm/hypervisor/xenserver/ PRE-CREATION 
  scripts/vm/hypervisor/xenserver/xenserver56/patch d485414 
  scripts/vm/hypervisor/xenserver/xenserver56fp1/patch 9fe9740 
  scripts/vm/hypervisor/xenserver/xenserver60/patch f049109 
  server/pom.xml cd4fbbe 
  server/src/com/cloud/api/ 3b5f634 
  server/src/com/cloud/api/ dfe4a1f 
  server/src/com/cloud/api/ ebe8415 
  server/src/com/cloud/api/doc/ d31ef5a 
  server/src/com/cloud/configuration/ 66ac276 
  server/src/com/cloud/configuration/ ef940e8 
  server/src/com/cloud/configuration/ ef61044 
  server/src/com/cloud/resource/ ced601b 
  server/src/com/cloud/servlet/ 2638c8b 
  server/src/com/cloud/storage/ e252633 
  server/src/com/cloud/storage/dao/ PRE-CREATION 
  server/src/com/cloud/storage/dao/ PRE-CREATION 
  server/src/com/cloud/storage/dao/ f5b6913 
  server/src/com/cloud/storage/dao/ 2a0dfc8 
  server/src/com/cloud/storage/dao/ PRE-CREATION 
  server/src/com/cloud/storage/dao/ PRE-CREATION 
  server/src/com/cloud/storage/s3/ PRE-CREATION 
  server/src/com/cloud/storage/s3/ PRE-CREATION 
  server/src/com/cloud/storage/snapshot/ a10298e 
  server/src/com/cloud/storage/snapshot/ 32e37e6 
  server/src/com/cloud/template/ PRE-CREATION 
  server/src/com/cloud/template/ 1e87de2 
  setup/db/create-schema.sql fff084e 
  setup/db/db/schema-40to410.sql PRE-CREATION 
  tools/apidoc/ eeaf2a2 
  tools/marvin/marvin/ bd8a5b2 
  tools/marvin/marvin/ bdf08cc 
  ui/dictionary.jsp b80e296 
  ui/scripts/cloudStack.js de3bd73 
  ui/scripts/sharedFunctions.js b6b3ef8 
  ui/scripts/system.js 9e3932f 
  ui/scripts/templates.js 74511f0 
  utils/pom.xml 2e4e74f 
  utils/src/com/cloud/utils/ be1627d 
  utils/src/com/cloud/utils/ PRE-CREATION 
  utils/src/com/cloud/utils/ 0f0ef05 
  utils/src/com/cloud/utils/db/ 7c1c943 
  utils/src/com/cloud/utils/nio/ 50e6a88 



I am submitting patch to begin the feedback process while we complete integration testing.
 I have verified that it does not interfere with normal CloudStack operations when S3-backed
Secondary Storage is disabled (the default setting) .  I have successfully tested operation
of single zone template and ISO scenarios on devcloud described in the design document.  I
am currently working through some issues in our multi-zone test environment to complete all
scenarios described.  The following are the known deficiencies of the current implementation
which I plan to correct in a subsequent patch:

   * Cross zone garbage collection: When a global asset is deleted from one zone's secondary
storage, it is not deleted from the secondary storage of other zones which have downloaded
it from the object store
   * S3 Configuration Update: The API only supports adding an object store configuration.
 Users should be able to edit the access key, secret key, connection timeout. max error retries,
and socket timeout.
   * Multi-threaded Uploads: Permit the upload of multiple assets to the object store simultaneously
to decrease the propagation latency across all zones.


S3 Configuration Form
S3 Enable Menu on the Zone Tab


John Burwell

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message