deltacloud-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Fojtik <mfoj...@redhat.com>
Subject Thread safety problems with Deltacloud
Date Thu, 18 Apr 2013 11:15:49 GMT
Hi,

I recently discovered an interesting bug, that occurs when you do a lot 
of parallel requests to Deltacloud API.

Let say you start Deltacloud API with the 'mock' driver as default 
driver. Then you do 3 parallel requests to retrieve RHEV-M images, 
realms and hardware_profiles. In that case I get this error:

<snip>
E, [2013-04-18T12:44:13.629135 #11892] ERROR -- 500: [LoadError] 
uninitialized constant Deltacloud::Drivers::Rhevm

/home/mfojtik/code/core/server/lib/deltacloud/helpers/driver_helper.rb:57:in 
`rescue in driver'
/home/mfojtik/code/core/server/lib/deltacloud/helpers/driver_helper.rb:54:in 
`driver'

127.0.0.1 - - [18/Apr/2013 12:44:13] "GET /api HTTP/1.1" rhevm 
http://provider.url 500 127410 0.1480
</snip>

Note, that when I do the curl request manually with same params 
(provider, accept, I get 200 back).

I think this might be a threading issue, but I'm not sure how to fix it.
I started looking at 'driver' method and what we do here is that we 
require the 'driver' source file and then return the initialized driver.

We do this for every request (except we don't 'require' the driver 
source if the driver class exists in current namespace).

My impression is that the 'require' method is not thread-safe. It can be 
demonstrated on this code:

begin
   driver_class
rescue NameError => e
   require_relative(driver_source_name) ? retry :
      raise(LoadError.new(e.message))
end

In this case, we try to return the 'driver_class' and if the constant 
does not exists, we require the driver source file and call 'retry' that 
will then try to return it again. I think, if you have multiple parallel 
requests, the 'require_relative' (which is just alias for 'require') 
behave incorrectly, because multiple threads are requiring the same file 
in parallel.

My fix for this, that so far works for me is to change the line after 
'rescue NameError => e' to:

Thread.exclusive { require_relative(driver_source_name) } ? retry : 
raise(LoadError.new(e.message))

With this I don't run into any problems with parallel requests (so far).
However, I'm not 'ruby threads' expert, so any advise from somebody more 
experienced is appreciated.

Also I think one task, we need to do in Deltacloud/CIMI in close future 
would be to identify spots in our code base that could be potentially 
not thread safe.

Having somebody to write some reasonable benchmarking tool (like 'ab' 
with different urls/drivers/etc) would help to identifying this spots.

   -- Michal

-- 

Michal Fojtik <mfojtik@redhat.com>
Deltacloud API, CloudForms

Mime
View raw message