incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanna Santhanam <Prasanna.Santha...@citrix.com>
Subject Re: Limits set for a resource can be exceeded
Date Thu, 12 Jul 2012 11:03:52 GMT
On Wed, Jul 11, 2012 at 05:50:47AM -0400, Devdeep Singh wrote:
> Hi,
> 
> I observed this issue while looking into how resource limiting
> works. When a request is placed to create a resource (instance,
> volume etc.) the check for resource limits and incrementing the
> resource count is done in the following order
> 
> 1. Check the limits will not be exceeded by the addition of another
> resource. Raise an exception if so.  2. Create a db entry for the
> resource.  3. Increment the resource count.
> 
> The check and increment of a resource are not synchronized to be
> atomic. This can lead to scenarios where two different requests can
> successfully check the limits for the same resource and increment
> the count beyond the limits on boundary conditions. 
> 
> For example: Suppose domain limits for a vm instance is set to 5 and
> there are already 4 instances created for the domain. Two requests
> are placed by two different users of the domain to create an
> instance.  1. User 1 request checks the limit. Verifies that domain
> limit of 5 will not be exceeded.  2. Similarly User 2 request checks
> the limit and verifies the domain limit of 5 will not be exceeded.
> This is because the first request still hasn't incremented the
> resource count.  3. Both the requests go ahead and increment the
> count (to 6) and exceed the limit.
> 
> I have observed this issue with instance creation. It may be present
> at other places too (volumes, public ips etc.). Before I go ahead
> and file a bug, I wanted to check if it is a valid concern or am I
> missing something.
> 

FWIW, this counter's atomicity used to be a real problem back in the
last few 2.2.y series of releases and partly the reason why the
updateResourceCount API came into being. But I remember this was fixed
for accounts limits. Whether domain-limits and other resource limits
follow the same increment/decrement code paths I'm not sure. If they
don't we most likely have the same bug.

Can we reproduce this trivially or will we have to hold the threads in
the debugger to cause this situation?

-- 
Prasanna.,

Mime
View raw message