cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-9626) Instance fails to start after unsuccesful compute offering upgrade.
Date Fri, 09 Dec 2016 19:58:59 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736187#comment-15736187
] 

ASF subversion and git services commented on CLOUDSTACK-9626:
-------------------------------------------------------------

Commit c89198f4c65e74a27a5586e4e18b0e71b164a904 in cloudstack's branch refs/heads/4.9 from
[~rohit.yadav@shapeblue.com]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=c89198f ]

Merge pull request #1796 from sudhansu7/CLOUDSTACK-9626

CLOUDSTACK-9626: Instance fails to start after unsuccesful computeISSUE
============
Instance fails to start after unsuccesful compute offering upgrade.

TROUBLESHOOTING
==================
We observed VM instance get compute values "cpuNumber","cpuSpeed","memory" removed from table
"user_vm_details", which cause instance fail to startup next time on XenServer

`mysql> select * from user_vm_details where vm_id=10;
--------------------------------------------------------------------------------------------------
id 	vm_id 	name 	value 	display

--------------------------------------------------------------------------------------------------
218 	10 	platform 	viridian:true;acpi:1;apic:true;pae:true;nx:true 	1
219 	10 	hypervisortoolsversion 	xenserver56 	1
220 	10 	Message.ReservedCapacityFreed.Flag 	true 	1

--------------------------------------------------------------------------------------------------
3 rows in set (0.00 sec)`

`2016-11-29 06:49:03,667 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-49c25b1d
job-125) (logid:114a2f1b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin
java.lang.NullPointerException
	at com.cloud.vm.UserVmManagerImpl.upgradeRunningVirtualMachine(UserVmManagerImpl.java:1664)
	at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1631)
	at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1561)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at com.sun.proxy.$Proxy197.upgradeVirtualMachine(Unknown Source)
	at org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin.execute(ScaleVMCmdByAdmin.java:48)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)`

REPRO STEPS
==================
1. Set global setting enable.dynamic.scale.vm to true
2. Create a custom Compute Offerings A
3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M
4. Create another custom Compute Offerings B
5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M (ensure 4 times
over previous memory size), then you will encounter scaling failed
6. Stop VM instance , you will never startup again

EXPECTED BEHAVIOR
==================
Succeed Startup VM instance

ACTUAL BEHAVIOR
==================
Fail to start instance

RCA:
The ROLLBACK does not take care of restoring old service offering details. In case failure
we are removing the new service offering details but restoring old service offering details
is missing.

Before Fix:
`user_vm_details before upgrade.
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id  | vm_id | name                               | value                               
           | display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 118 |     9 | platform                           | viridian:true;acpi:1;apic:true;pae:true;nx:true
|       1 |
| 119 |     9 | hypervisortoolsversion             | xenserver56                         
           |       1 |
| 120 |     9 | Message.ReservedCapacityFreed.Flag | false                               
           |       1 |
| 121 |     9 | cpuNumber                          | 1                                   
           |       1 |
| 122 |     9 | cpuSpeed                           | 1000                                
           |       1 |
| 123 |     9 | memory                             | 256                                 
           |       1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
6 rows in set (0.00 sec)

user_vm_details after unsuccessful upgrade.
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id  | vm_id | name                               | value                               
           | display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 133 |     9 | platform                           | viridian:true;acpi:1;apic:true;pae:true;nx:true
|       1 |
| 134 |     9 | hypervisortoolsversion             | xenserver56                         
           |       1 |
| 135 |     9 | Message.ReservedCapacityFreed.Flag | false                               
           |       1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
3 rows in set (0.00 sec)`

After fix:
`
mysql> select * from user_vm_details where vm_id =9;
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| id  | vm_id | name                               | value                               
           | display |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
| 166 |     9 | cpuNumber                          | 1                                   
           |       1 |
| 167 |     9 | platform                           | viridian:true;acpi:1;apic:true;pae:true;nx:true
|       1 |
| 168 |     9 | cpuSpeed                           | 1000                                
           |       1 |
| 169 |     9 | Message.ReservedCapacityFreed.Flag | false                               
           |       1 |
| 170 |     9 | memory                             | 256                                 
           |       1 |
| 171 |     9 | hypervisortoolsversion             | xenserver56                         
           |       1 |
+-----+-------+------------------------------------+-------------------------------------------------+---------+
6 rows in set (0.00 sec)
`

* pr/1796:
  CLOUDSTACK-9626: Instance fails to start after unsuccesful compute offering  upgrade.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>


> Instance fails to start after unsuccesful compute offering upgrade.
> -------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-9626
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9626
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.8.0
>            Reporter: Sudhansu Sahu
>            Assignee: Sudhansu Sahu
>
> ISSUE
> ============
> Instance fails to start after unsuccesful compute offering upgrade.
>  
> TROUBLESHOOTING
> ==================
> We observed VM instance get compute values "cpuNumber","cpuSpeed","memory" removed from
table "user_vm_details", which cause instance fail to startup next time on XenServer 
> mysql> select * from user_vm_details where vm_id=10;
> +-----+-------+------------------------------------+-------------------------------------------------+---------+
> | id  | vm_id | name                               | value                          
                | display |
> +-----+-------+------------------------------------+-------------------------------------------------+---------+
> | 218 |    10 | platform                           | viridian:true;acpi:1;apic:true;pae:true;nx:true
|       1 |
> | 219 |    10 | hypervisortoolsversion             | xenserver56                    
                |       1 |
> | 220 |    10 | Message.ReservedCapacityFreed.Flag | true                           
                |       1 |
> +-----+-------+------------------------------------+-------------------------------------------------+---------+
> 3 rows in set (0.00 sec)
>  
> Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.ScaleVMCmd
> java.lang.NullPointerException
> 	at com.cloud.vm.UserVmManagerImpl.upgradeStoppedVirtualMachine(UserVmManagerImpl.java:953)
> 	at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1331)
> 	at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1271)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:616)
> 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> 	at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> 	at $Proxy169.upgradeVirtualMachine(Unknown Source)
> 	at org.apache.cloudstack.api.command.user.vm.ScaleVMCmd.execute(ScaleVMCmd.java:127)
> 	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:167)
> 	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:97)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:543)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:50)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> 	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> 	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:47)
> 	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:679)
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Select by Id SQL:
SELECT user_vm_details.id, user_vm_details.vm_id, user_vm_details.name, user_vm_details.value,
user_vm_details.display FROM user_vm_details WHERE user_vm_details.id = ? 
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Table References:
user_vm_details
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Insert SQLs:
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) INSERT INTO user_vm_details
(user_vm_details.vm_id, user_vm_details.name, user_vm_details.value, user_vm_details.display)
VALUES (?, ?, ?, ?)
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Delete SQLs
> 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) DELETE FROM user_vm_details
WHERE user_vm_details.id= ?
> REPRO STEPS
> ==================
> 1. Set global setting enable.dynamic.scale.vm to true
> 2. Create a custom  Compute Offerings A
> 3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M
> 4. Create another custom Compute Offerings B
> 5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M (ensure 4
times over previous memory size), then you will encounter scaling failed
> 6. Stop VM instance , you will never startup again
> EXPECTED BEHAVIOR
> ==================
> Succeed Startup VM instance
>  
> ACTUAL BEHAVIOR
> ==================
> Fail to start instance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message