incubator-deltacloud-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Fojtik <mfoj...@redhat.com>
Subject Re: [PATCH] Brings mock driver and yml up-to-date with the blobstore (incl. metadata)
Date Mon, 14 Feb 2011 15:44:42 GMT
On 10/02/11 21:30 +0200, marios@redhat.com wrote:

ACK. Looks good, I tested it manually using RestClient and works well, just
small code comments bellow.

Btw. Do we have some Test::Unit files for this in place?

  -- Michal


>From: marios <marios@redhat.com>
>
>---
> .../drivers/mock/data/buckets/blobs/blob1.yml      |    9 +-
> .../drivers/mock/data/buckets/blobs/blob2.yml      |    6 +-
> .../drivers/mock/data/buckets/blobs/blob3.yml      |    6 +-
> .../drivers/mock/data/buckets/blobs/blob4.yml      |    7 +-
> .../drivers/mock/data/buckets/blobs/blob5.yml      |    6 +-
> server/lib/deltacloud/drivers/mock/mock_driver.rb  |  114 ++++++++++++++++----
> .../drivers/rackspace/rackspace_driver.rb          |    4 +-
> server/lib/deltacloud/helpers/blob_stream.rb       |    1 +
> server/server.rb                                   |   12 ++-
> 9 files changed, 129 insertions(+), 36 deletions(-)
>
>diff --git a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob1.yml b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob1.yml
>index 5ead10f..cf119bc 100644
>--- a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob1.yml
>+++ b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob1.yml
>@@ -1,5 +1,8 @@
>-:bucket: bucket1
>-:content_length: 12627
>+---
> :content_type: text/plain
>+:content_length: 17
>+:bucket: bucket1
>+:user_metadata:
>+  SOMENEWKEY: NEWVALUE
> :last_modified: 2010-09-23 16:44:54 +0100
>-:content: "content of blob 1"
>\ No newline at end of file
>+:content: content of blob 1
>diff --git a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob2.yml b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob2.yml
>index 4e62dd0..cb3e36d 100644
>--- a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob2.yml
>+++ b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob2.yml
>@@ -1,5 +1,7 @@
> :bucket: bucket1
>-:content_length: 4420
>+:content_length: 56
> :content_type: text/html
> :last_modified: 2010-09-23 16:55:05 +0100
>-:content: "content of blob 2"
>\ No newline at end of file
>+:content:
>+    <html><head></head>content of blob 2<body></body></html>
>+:user_metadata: ''
>diff --git a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob3.yml b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob3.yml
>index 31cc1b3..f0a4d12 100644
>--- a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob3.yml
>+++ b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob3.yml
>@@ -1,5 +1,7 @@
> :bucket: bucket1
>-:content_length: 98732
>+:content_length: 17
> :content_type: text/plain
> :last_modified: 2010-08-14 02:14:31 +0100
>-:content: "content of blob 3"
>\ No newline at end of file
>+:content:
>+    content of blob 3
>+:user_metadata: ''
>\ No newline at end of file
>diff --git a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob4.yml b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob4.yml
>index c9873ac..0dbeef1 100644
>--- a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob4.yml
>+++ b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob4.yml
>@@ -1,5 +1,8 @@
> :bucket: bucket2
>-:content_length: 983232
>+:content_length: 17
> :content_type: application/octet-stream
> :last_modified: 2010-09-21 06:17:24 +0100
>-:content: "content of blob 4"
>\ No newline at end of file
>+:content:
>+    content of blob 4
>+:user_metadata:
>+    some: "value"
>\ No newline at end of file
>diff --git a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob5.yml b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob5.yml
>index 5014c89..fac8fce 100644
>--- a/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob5.yml
>+++ b/server/lib/deltacloud/drivers/mock/data/buckets/blobs/blob5.yml
>@@ -1,5 +1,7 @@
> :bucket: bucket2
>-:content_length: 651922
>+:content_length: 17
> :content_type: application/pdf
> :last_modified: 2009-07-21 11:12:13 +0100
>-:content: "content of blob 5"
>\ No newline at end of file
>+:content:
>+    content of blob 5
>+:user_metadata: ''
>\ No newline at end of file
>diff --git a/server/lib/deltacloud/drivers/mock/mock_driver.rb b/server/lib/deltacloud/drivers/mock/mock_driver.rb
>index a36d569..de16c8b 100644
>--- a/server/lib/deltacloud/drivers/mock/mock_driver.rb
>+++ b/server/lib/deltacloud/drivers/mock/mock_driver.rb
>@@ -321,14 +321,21 @@ class MockDriver < Deltacloud::BaseDriver
>   def buckets(credentials, opts={})
>     check_credentials(credentials)
>     buckets=[]
>-     Dir[ "#{@storage_root}/buckets/*.yml" ].each do |bucket_file|

What if this directory doesn't exists ? It's created in Rake task? (rake fixtures?)

>-      bucket = YAML.load( File.read( bucket_file ) )
>-      bucket[:id] = File.basename( bucket_file, ".yml" )
>-      bucket[:name] = bucket[:id]
>-      buckets << Bucket.new( bucket )
>+    safely do
>+      unless (opts[:id].nil?)
>+        bucket_file = "#{@storage_root}/buckets/#{opts[:id]}.yml"

Maybe it would be better to use File::join method, because it will use the
right file delimiter (\ for Windows). I bet we are using this on many
places, so maybe it will worth to do some code audit.

File::join(@storage_root, buckets, "#{opts[:id]}.yml")


>+        bucket = YAML.load( File.read( bucket_file ) )
>+        bucket[:id] = opts[:id]
>+        bucket[:name] = bucket[:id]
>+        buckets << Bucket.new( bucket )
>+      else
>+         Dir[ "#{@storage_root}/buckets/*.yml" ].each do |bucket_file|
>+          bucket_id = File.basename( bucket_file, ".yml" )
>+          buckets << Bucket.new( {:id => bucket_id, :name => bucket_id }
)
>+        end
>+      end
>     end
>     buckets = filter_on( buckets, :id, opts )
>-    buckets
>   end
>
> #--
>@@ -350,6 +357,7 @@ class MockDriver < Deltacloud::BaseDriver
> # Delete bucket
> #--
>   def delete_bucket(credentials, name, opts={})
>+    check_credentials(credentials)
>     bucket = bucket(credentials, {:id => name})
>     unless (bucket.size == "0")
>      raise Deltacloud::BackendError.new(403, self.class.to_s, "bucket-not-empty", "delete
operation not valid for non-empty bucket")
>@@ -365,14 +373,15 @@ class MockDriver < Deltacloud::BaseDriver
>   def blobs(credentials, opts = {})
>     check_credentials(credentials)
>     blobs=[]
>-    Dir[ "#{@storage_root}/buckets/blobs/*.yml" ].each do |blob_file|
>-      blob = YAML.load( File.read( blob_file ) )
>-      blob[:id] = File.basename( blob_file, ".yml" )
>+    blobfile = "#{@storage_root}/buckets/blobs/#{opts[:id]}.yml"
>+    safely do
>+      blob = YAML.load( File.read( blobfile ) )
>+      return [] unless blob[:bucket] == opts['bucket'] #can't return nil since base_driver
invokes .first on return
>+      blob[:id] = File.basename( blobfile, ".yml" )
>       blob[:name] = blob[:id]
>       blobs << Blob.new( blob )
>+      blobs = filter_on( blobs, :id, opts )
>     end
>-    blobs = filter_on( blobs, :id, opts )
>-    blobs
>   end
>
> #--
>@@ -381,11 +390,73 @@ class MockDriver < Deltacloud::BaseDriver
>   def blob_data(credentials, bucket_id, blob_id, opts = {})
>     check_credentials(credentials)
>     blob=nil
>-    Dir[ "#{@storage_root}/buckets/blobs/*.yml" ].each do |blob_file|
>-      if File.basename(blob_file, ".yml") == blob_id
>-        blob = YAML.load(File.read(blob_file))
>-        blob[:content].each {|part| yield part}
>+    safely do
>+      blobfile = "#{@storage_root}/buckets/blobs/#{opts['blob']}.yml"
>+      blob = YAML.load(File.read(blobfile))

You can do: YAML.load_file(blobfile) 

http://santoro.tk/mirror/ruby-core/classes/YAML.html#M007377

>+    end
>+    blob[:content].each {|part| yield part}
>+  end
>+
>+#--
>+# Create blob
>+#--
>+  def create_blob(credentials, bucket_id, blob_id, blob_data, opts={})
>+    check_credentials(credentials)
>+      blob_meta = {}
>+      opts.inject({}){|result, (k,v)| blob_meta[k] = v if k.match(/X[_-]Deltacloud[_-]Blobmeta[_-]/i)}
#select{|k,v| k.match(/X[_-]Deltacloud[_-]Blobmeta[_-]/i)}
>+       blob = {
>+      :id => blob_id,
>+      :bucket => bucket_id,
>+      :content_length => blob_data[:tempfile].length,
>+      :content_type => blob_data[:type],
>+      :last_modified => Time.now,
>+      :user_metadata => blob_meta.gsub_keys('X_Deltacloud_Blobmeta_', ''),
>+      :content => blob_data[:tempfile].read
>+    }
>+    File.open( "#{@storage_root}/buckets/blobs/#{blob_id}.yml", 'w' ) {|b| YAML.dump(
blob, b )}
>+    Blob.new(blob)
>+  end
>+
>+#--
>+# Delete blob
>+#--
>+  def delete_blob(credentials, bucket_id, blob_id, opts={})
>+    check_credentials(credentials)
>+    blobfile = "#{@storage_root}/buckets/blobs/#{blob_id}.yml"
>+    safely do
>+      unless File.exists?(blobfile)
>+        raise Deltacloud::BackendError.new(500, self.class.to_s, "blob #{blob_id} doesn't
exist", "cannot delete non existant blob")
>       end
>+      File.delete(blobfile)
>+    end
>+  end
>+
>+#--
>+# Get metadata
>+#--
>+  def blob_metadata(credentials, opts={})
>+    check_credentials(credentials)
>+    blobfile = "#{@storage_root}/buckets/blobs/#{opts[:id]}.yml"
>+    #safely do - mechanism not suitable here since head requests don't return a body
response
>+    begin
>+      blob = YAML.load(File.read(blobfile))
>+    rescue Errno::ENOENT
>+      return nil #server.rb picks this up and gives 404

Maybe throwing an Exception is more OO way how to indicate this ;-)

>+    end
>+    blob[:user_metadata]
>+  end
>+
>+#--
>+# Update metadata
>+#--
>+  def update_blob_metadata(credentials, opts={})
>+    check_credentials(credentials)
>+    blobfile = "#{@storage_root}/buckets/blobs/#{opts[:id]}.yml"
>+    safely do
>+      blob = YAML.load(File.read(blobfile))
>+      return false unless blob
>+      blob[:user_metadata] = opts['meta_hash'].gsub_keys('HTTP[-_]X[-_]Deltacloud[-_]Blobmeta[-_]',
'')
>+      File.open( "#{@storage_root}/buckets/blobs/#{opts[:id]}.yml", 'w' ) {|b| YAML.dump(
blob, b )}
>     end
>   end
>
>@@ -401,15 +472,18 @@ class MockDriver < Deltacloud::BaseDriver
>   private
>
>   def check_credentials(credentials)
>-    if ( credentials.user != 'mockuser' )
>-      raise Deltacloud::AuthException.new
>-    end
>-
>-    if ( credentials.password != 'mockpassword' )
>+    if ( credentials.user != 'mockuser' ) or ( credentials.password != 'mockpassword'
)
>       raise Deltacloud::AuthException.new
>     end
>   end
>
>+  def catched_exceptions_list
>+    {
>+      :auth => [],
>+      :error => [ ::Deltacloud::BackendError, Errno::ENOENT ],
>+      :glob => []
>+    }
>+  end
>
> end
>
>diff --git a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
>index 8d743aa..a892f34 100644
>--- a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
>+++ b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
>@@ -259,7 +259,9 @@ class RackspaceDriver < Deltacloud::BaseDriver
> #--
>   def delete_blob(credentials, bucket_id, blob_id, opts={})
>     cf = cloudfiles_client(credentials)
>-    cf.container(bucket_id).delete_object(blob_id)
>+    safely do
>+      cf.container(bucket_id).delete_object(blob_id)
>+    end
>   end
>
> #-
>diff --git a/server/lib/deltacloud/helpers/blob_stream.rb b/server/lib/deltacloud/helpers/blob_stream.rb
>index a0dfd42..0d0a4be 100644
>--- a/server/lib/deltacloud/helpers/blob_stream.rb
>+++ b/server/lib/deltacloud/helpers/blob_stream.rb
>@@ -15,6 +15,7 @@
> # License for the specific language governing permissions and limitations
> # under the License.
>
>+include Deltacloud
> begin
>   require 'eventmachine'
>   #--
>diff --git a/server/server.rb b/server/server.rb
>index 3f092e4..61d9c91 100644
>--- a/server/server.rb
>+++ b/server/server.rb
>@@ -699,10 +699,14 @@ end
> #get the content of a particular blob
> get '/api/buckets/:bucket/:blob/content' do
>   @blob = driver.blob(credentials, { :id => params[:blob], 'bucket' => params[:bucket]})
>-  params['content_length'] = @blob.content_length
>-  params['content_type'] = @blob.content_type
>-  params['content_disposition'] = "attachment; filename=#{@blob.id}"
>-  BlobStream.call(env, credentials, params)
>+  if @blob
>+    params['content_length'] = @blob.content_length
>+    params['content_type'] = @blob.content_type
>+    params['content_disposition'] = "attachment; filename=#{@blob.id}"
>+    BlobStream.call(env, credentials, params)
>+  else
>+    report_error(404, 'not_found')
>+  end
> end
>
> #Get html form for creating a new bucket
>--
>1.7.3.4
>

-- 
--------------------------------------------------------
Michal Fojtik, mfojtik@redhat.com
Deltacloud API: http://deltacloud.org
--------------------------------------------------------

Mime
View raw message