incubator-deltacloud-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject Re: Adds blob streaming uploads using PUT - REVISION 1
Date Wed, 08 Jun 2011 07:21:10 GMT
sorry, forgot to mention, this code requires sinatra >= 1.2.4

In cases where the streaming monkey patch is not executed ('PUT' blobs 
that are smaller than 112 KB or when running webrick instead of thin) I 
trigger the 'POST' route from the 'PUT' route, as explained in the 
sinatra docs " Another Route".

There was an issue with this functionality that was fixed in sinatra 
1.2.4 - I'll update the deltacloud-core.gemspec (line 64: 
s.add_dependency('sinatra', '>= 1.2.4') ) accordingly when this is going 
to be pushed - unless anyone has objections/issues with using this 
version of sinatra,

thanks, marios

On 07/06/11 20:45, wrote:
> * incorporates comments by David Lutterkort in response to v1 (date: patch on 20 May
2011, comments on 24 May, Subject: "Adds blob streaming uploads using PUT (client ---STREAM--->
 deltacloud ---STREAM--->  provider)" )
> * handles blob metadata using the usual X-Deltacloud-Blobmeta-KEY:VALUE headers
> * moves connection code into a driver method (ec2_driver: 'def blob_stream_connection')
> * fixes issue with using PUT and files smaller than 112KB (whereby streaming is not triggered
so we use a tempfile - ala the POST/html interface)
> * webrick safe - detects if thin is being used and if not disables streaming put and
triggers the post method
> marios
> =======================================================
> Message from original patch copied here for convenience:
> Streaming for uploading blobs to provider. Currently to create a blob we receive the
blob from the client, load it into memory and then stream it up to the provider. i.e.:
> client ---TEMP_FILE--->  deltacloud ---STREAM--->  provider
> Obviously this is not very useful for large blobs (e.g. an image)  - especially when
there can be any number of 'create blob' requests which the server would have to create temp_files
> You can try it out using
> curl -H 'content-type: text/html' -H 'X-Deltacloud-Blobmeta-Name:mariosblob' -H 'X-Deltacloud-Blobmeta-Version:niceone'
-H 'X-Deltacloud-Blobmeta-Author:me' -iv --upload-file "/home/marios/Desktop/larger.txt" 
--user 'awskey:aws_secretkey'  http://localhost:3001/api/buckets/thebucket_name/the_new_blob_name?format=xml
> Since Net::HTTP doesn't provide a way for us to do this 'nicely' the only way to go was
with monkey patching (both to Thin::Request and Net::HTTP). Basically we hook into the 'receive'
method of the thin server so as the data is read in from the client, it starts getting sent
out to the provider. Also, aws gem (appoxy) doesn't provide this functionality so I do the
PUT directly using (a monkey patched version of) Net::HTTP. At this point this is an ec2 specific
solution - only because its rev. 1 - we will add other providers if this seems like a sane
way to proceed.
> Also, since theres no way to do a 'put' using browser (the sinatra 'method override'
stuff is not helpful here since that code kicks in way later than where I am starting to stream)
I leave the original 'POST /api/buckets/:bucket' rule in there as an alternative (i.e. using
the html interface and a HTTP multipart/form data upload) though this is non streaming.

View raw message