cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-8603) Random list VM failures at scale (more than 1000 VMs) when VM has resource tags
Date Fri, 03 Jul 2015 09:59:04 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613080#comment-14613080
] 

ASF subversion and git services commented on CLOUDSTACK-8603:
-------------------------------------------------------------

Commit 7eac6310bc1f67e8c61980d43cd31586032503ee in cloudstack's branch refs/heads/4.5 from
[~koushikd]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=7eac631 ]

CLOUDSTACK-8603: Random list VM failures at scale (more than 1000 VMs) when VM has resource
tags There is no 'removed' field on the resource_tags table. So 'id' based search may return
a record or null in case record is deleted. Added a check for null or empty in search resource
tags based on 'id'.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

This closes #551

(cherry picked from commit 5d9f851deb1a12e4a07297353ea715f9cf13e677)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>


> Random list VM failures at scale (more than 1000 VMs) when VM has resource tags
> -------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-8603
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8603
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.5.0, 4.5.1
>            Reporter: Koushik Das
>            Assignee: Koushik Das
>             Fix For: 4.6.0
>
>
> Random failures seen in list VM API call (VMs having resource tag) with the following
exception. As can be seen from the stack trace, exception is thrown while creating the list
response. The test scenario involved doing VM creation/deletion/list operations concurrently
over a period of time.
> 2015-05-27 00:16:54,239 ERROR [c.c.a.ApiServer] (catalina-exec-24:ctx-c122d00c ctx-fcdc0d00
ctx-3df89f23) (logid:6004902e) unhandled exception executing api command: [Ljava.lang.String;@76642f16
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at com.cloud.api.query.dao.ResourceTagJoinDaoImpl.searchById(ResourceTagJoinDaoImpl.java:154)
> at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy86.searchById(Unknown Source)
> at com.cloud.api.ApiDBUtils.findResourceTagViewById(ApiDBUtils.java:1636)
> at com.cloud.api.query.dao.UserVmJoinDaoImpl.newUserVmResponse(UserVmJoinDaoImpl.java:266)
> at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy176.newUserVmResponse(Unknown Source)
> at com.cloud.api.ApiDBUtils.newUserVmResponse(ApiDBUtils.java:1600)
> at com.cloud.api.query.ViewResponseHelper.createUserVmResponse(ViewResponseHelper.java:140)
> at com.cloud.api.query.QueryManagerImpl.searchForUserVMs(QueryManagerImpl.java:762)
> at org.apache.cloudstack.api.command.user.vm.ListVMsCmd.execute(ListVMsCmd.java:227)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:700)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:525)
> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:282)
> There are 2 part to list VM API:
> a. Get a list of VMs based on search criteria from the user_vm_view view in DB.
> b. Create the response records corresponding to each VM obtained as part of (a). This
record creation involves fetching additional data from other views/tables in the DB. One such
view is resource_tag_view.
> Now consider the following sequence of events that may lead to the issue:
> a. List VM call fetches a lot of VMs from user_vm_view.
> b. VM response is created for each VM by iterating over the list obtained from (a).
> c. Now while (b) is in progress, a VM which is in the list from (a) and not yet processed
in (b) got destroyed.
> d. Now when response generation happens for destroyed VM, there is a DB query to fetch
resource tag details from resource_tag_view. This query returns empty as the resource tag
got deleted from DB as part of VM destroy.
> e. In the resource tag fetching code there is no check for the condition mentioned in
(d) and so the index out of bound exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message