incubator-deltacloud-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject Adds blob streaming uploads using PUT - REVISION 2
Date Wed, 08 Jun 2011 17:09:31 GMT

* incorporates comments by David Lutterkort in response to Revision 1 (date: patch on 07 June
2011, comments on 08 June, Subject: "Adds blob streaming uploads using PUT - REVISION 1" )
- Patch 1

* removes the necessity for sinatra >= 1.2.4 (no longer calling the POST blob route from
PUT blob)

* moves formatting nits into seperate patches (patches 2+3)

* moves extraction of blob metadata headers, and renaming of headers per provider, into helpers
methods (patch 4)


Message from original patch copied here for convenience:

Streaming for uploading blobs to provider. Currently to create a blob we receive the blob
from the client, load it into memory and then stream it up to the provider. i.e.:

client ---TEMP_FILE---> deltacloud ---STREAM---> provider

Obviously this is not very useful for large blobs (e.g. an image)  - especially when there
can be any number of 'create blob' requests which the server would have to create temp_files

You can try it out using

curl -H 'content-type: text/html' -H 'X-Deltacloud-Blobmeta-Name:mariosblob' -H 'X-Deltacloud-Blobmeta-Version:niceone'
-H 'X-Deltacloud-Blobmeta-Author:me' -iv --upload-file "/home/marios/Desktop/larger.txt" 
--user 'awskey:aws_secretkey'  http://localhost:3001/api/buckets/thebucket_name/the_new_blob_name?format=xml

Since Net::HTTP doesn't provide a way for us to do this 'nicely' the only way to go was with
monkey patching (both to Thin::Request and Net::HTTP). Basically we hook into the 'receive'
method of the thin server so as the data is read in from the client, it starts getting sent
out to the provider. Also, aws gem (appoxy) doesn't provide this functionality so I do the
PUT directly using (a monkey patched version of) Net::HTTP. At this point this is an ec2 specific
solution - only because its rev. 1 - we will add other providers if this seems like a sane
way to proceed. 

Also, since theres no way to do a 'put' using browser (the sinatra 'method override' stuff
is not helpful here since that code kicks in way later than where I am starting to stream)
I leave the original 'POST /api/buckets/:bucket' rule in there as an alternative (i.e. using
the html interface and a HTTP multipart/form data upload) though this is non streaming.

View raw message