Mailing-List: contact deltacloud-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: deltacloud-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of jdarcy@redhat.com designates
 209.132.183.28 as permitted sender)
Message-ID: <4C3E61C4.8080703@redhat.com>
Date: Wed, 14 Jul 2010 21:17:56 -0400
From: Jeff Darcy <jdarcy@redhat.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4
MIME-Version: 1.0
CC: deltacloud-dev@incubator.apache.org
Subject: Re: Cloud Storage and Image upload - Instance snapshot via deltacloud
 core API
References: <4C3B420F.5050607@redhat.com>
In-Reply-To: <4C3B420F.5050607@redhat.com>
Content-Type: multipart/alternative;
 boundary="------------090100050002040405000208"

--------------090100050002040405000208
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 01/-10/-28163 02:59 PM, marios@redhat.com wrote:
> I think there are two independent subtasks here:
>
> **Work out the operations for the generic cloud storage (i.e.
> s3/cloudfiles) and implement 'Blob Management'. This will appear as a
> top level collection (like Instances, Images etc, 'Blobs' or somesuch
> name 'Objects'?) - to appear obviously only for those drivers that
> support it, Amazon AWS and Rackspace.

Also any cloud running the S3-compatible stack from Project Hail.

> **Implement the Instance snapshot function for those clouds where it
> makes sense (e.g. do we want to implement the 'clone this Instance' type
> functionality offered by Terremark and Rimuhosting?). This could be
> exposed as an Instance 'action' (like stop, start, reboot) ?

I think it's good to consider these as two separate functional areas, 
but I think we also want to preserve the relationships between them 
wherever applicable.  An image might be an object unto itself, or it 
might be a reference (named or unnamed) to another object (block or blob 
storage).  Whenever possible, the image repo should be able to deposit 
an image using the block/blob API (e.g. EBS or S3 respectively for EC2) 
but also maintain awareness of the name (e.g. AMI ID) that would be used 
for the same image in the instance-creation API.  In the case where an 
image is a completely separate type of object we'd have to stub out the 
data object but hopefully can still attach all of the tags/metadata that 
the repo uses to keep track of images.

> =====================================================
> Q. Which of our currently supported clouds offers a generic 'upload
> data/blob/object to our storage service and access from anywhere even
> from machines outside our cloud' - type storage?
> =====================================================
>
> --i--> Amazon Web Services (AWS) offers the Simple Storage Service (S3)
> [1] - not to be confused with S4 [2]

We might also need to account for EBS-based AMIs, especially because of 
the S3 size limitation (sizes are unlimited in tabled BTW).

> --ii-> Rackspace offers CloudFiles [3]
>
>     ** The services are very similar.
>     ** Create 'buckets' in S3 and 'containers' is CloudFiles, and place
> 'objects' within these.
>     ** Each S3 account is limited to 100 buckets, each holding an
> unlimited number of objects with each at most 5 GBytes.
>     ** A CloudFiles account can have unlimited buckets and objects;
> again each object must be < 5 GBytes.
>
> =====================================================
> Q. Which of our currently supported clouds allows 'register custom Image
> from cloud storage', or, 'save/snapshot an Instance as a new Image'
> =====================================================
>
> --i--> Amazon AWS allows you to register an object in S3 as an Image in
> EC2 [4] (Rackspace 'coming soon' [5])
>
>     ** The process of 'save this Instance as an Image in S3' is termed
> 'bundling'
>     ** PROBLEM: different procedure for Windose vs Linux images [6].
>     -- For Windows you use the AWS REST API 'BundleInstance' call 
> where you specify the S3 bucket to upload to (amongst other things).
>     -- For Linux you must install the 'AMI Tools' command-line 
> utilities [7] on the Instance you wish to bundle and then 
> 'ec2-bundle-image'. Confusingly the actual upload is done with 
> 'ec2-upload-bundle' which takes the S3 bucket name as a parameter.
>
>     ** In both cases and after bundling is complete you register the
> newly created S3 object as an Image for EC2 using the REST API
> 'RegisterImage' call.

The steps for creating a new EBS-backed image from a running S3-backed 
instance, and for creating one EBS-backed image from another, seem 
substantially similar.

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?Using_BootFromEBS.html

>
> =====================================================
>
> --ii-> Rackspace allows you to snapshot a running Instance [5]
>
>     ** This is done through a REST API call (POST /images) where you
> specify the serverid (Instance) that is to be saved.
>     ** Here there is no seperation between 'make snapshot' and 'upload
> to cloud storage' - i.e. cannot export a server image [8].
>     ** Snapshots stay 'attached to the server' [9] (i.e. saved locally?).
>     ** Once process complete the image is available for booting a server
> [5].
>     ** 'Custom Image Upload' is in the Cloud Server API Roadmap [5].

Once you've POSTed an image, is that image retrievable using CloudFiles, 
or are images totally stranded in a separate namespace from other objects?
>
> =====================================================
>
> -iii-> Gogrid allows you to save a 'MyGSI - Gogrid Server Image' from
> your own, preconfigured 'sandbox server'
>
>     ** You create a MyGSI from a configured, running 'image sandbox
> server' and the new image is added to Gogrid's cloud storage [11]
>     ** Weirdly this cloud storage is not exposed at all via the API
>     ** The API call is grid.image.save [12] - you create a new 'image
> sandbox server' using 'grid.server.add', configure it and run the 'prep
> scripts' and then call the save. Once complete the new MyGSI is
> available in your list of images.

I've also found this process less than reliable.  To be more precise, 
every time I tried it just sat there for half an hour or so and then 
threw up.

>
> =====================================================
>
> --iv-> Rimuhosting allows you to clone an existing server and deploy a
> new server from that [10]
>
>     ** However, the clone must be deployed straight away - you cannot
> save as an image for later instantiation.
>     ** A chat with support confirmed this - you *can* save an image for
> later use but only by submitting a support ticket (i.e. nothing in the
> api or the web 'management console' to support this).
>
> =====================================================
>
> --v--> Terremark VCloud Express allows you to 'clone' an existing server
> (a 'vapp') [13] but you cannot save this as an Image (a 'vapp template').
>
>     ** Similar to the rimuhosting situation - can clone an existing
> server as another server, but not as an Image
>     ** POST https://{Terremark URI}/vdc/{vDC ID}/action/clonevapp - you
> specify the vApp id to be cloned within the request body.
>
> =====================================================
>
> AWS:
> [1] http://aws.amazon.com/documentation/s3/
> [2] http://www.supersimplestorageservice.com/
> [4] http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/
> [6]
> http://developer.amazonwebservices.com/connect/thread.jspa?messageID=107847&#107847 
>
> [7]
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88 
>
> RACKSPACE:
> [3] http://www.rackspacecloud.com/cloud_hosting_products/files/api
> [5] http://docs.rackspacecloud.com/servers/api/cs-devguide-latest.pdf
> [8] http://www.rackspacecloud.com/cloud_hosting_products/servers/faq
> [9] http://www.rackspacecloud.com/cloud_hosting_products/servers/pricing
> RIMUHOSTING:
> [10]
> http://apidocs.rimuhosting.com/jaxbdocs/com/rimuhosting/jaxrsorders/JAXRSOrders.NewVPSRequest.html 
>
> GOGRID:
> [11] http://wiki.gogrid.com/wiki/index.php/Cloud_Storage:Cloud_Storage
> [12] http://wiki.gogrid.com/wiki/index.php/API:grid.image.save
> TERREMARK:
> [13]
> https://community.vcloudexpress.terremark.com/en-us/product_docs/w/wiki/6-vcloud-express-api-documention.aspx 
>
>


--------------090100050002040405000208--