Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6530E200BFC for ; Fri, 9 Dec 2016 20:59:02 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 64443160B1E; Fri, 9 Dec 2016 19:59:02 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1746A160B1D for ; Fri, 9 Dec 2016 20:59:00 +0100 (CET) Received: (qmail 31297 invoked by uid 500); 9 Dec 2016 19:58:59 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 31137 invoked by uid 500); 9 Dec 2016 19:58:59 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 31116 invoked by uid 99); 9 Dec 2016 19:58:59 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Dec 2016 19:58:59 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 992122C03DE for ; Fri, 9 Dec 2016 19:58:59 +0000 (UTC) Date: Fri, 9 Dec 2016 19:58:59 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-9626) Instance fails to start after unsuccesful compute offering upgrade. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 09 Dec 2016 19:59:02 -0000 [ https://issues.apache.org/jira/browse/CLOUDSTACK-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736187#comment-15736187 ] ASF subversion and git services commented on CLOUDSTACK-9626: ------------------------------------------------------------- Commit c89198f4c65e74a27a5586e4e18b0e71b164a904 in cloudstack's branch refs/heads/4.9 from [~rohit.yadav@shapeblue.com] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=c89198f ] Merge pull request #1796 from sudhansu7/CLOUDSTACK-9626 CLOUDSTACK-9626: Instance fails to start after unsuccesful computeISSUE ============ Instance fails to start after unsuccesful compute offering upgrade. TROUBLESHOOTING ================== We observed VM instance get compute values "cpuNumber","cpuSpeed","memory" removed from table "user_vm_details", which cause instance fail to startup next time on XenServer `mysql> select * from user_vm_details where vm_id=10; -------------------------------------------------------------------------------------------------- id vm_id name value display -------------------------------------------------------------------------------------------------- 218 10 platform viridian:true;acpi:1;apic:true;pae:true;nx:true 1 219 10 hypervisortoolsversion xenserver56 1 220 10 Message.ReservedCapacityFreed.Flag true 1 -------------------------------------------------------------------------------------------------- 3 rows in set (0.00 sec)` `2016-11-29 06:49:03,667 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-49c25b1d job-125) (logid:114a2f1b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin java.lang.NullPointerException at com.cloud.vm.UserVmManagerImpl.upgradeRunningVirtualMachine(UserVmManagerImpl.java:1664) at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1631) at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1561) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy197.upgradeVirtualMachine(Unknown Source) at org.apache.cloudstack.api.command.admin.vm.ScaleVMCmdByAdmin.execute(ScaleVMCmdByAdmin.java:48) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)` REPRO STEPS ================== 1. Set global setting enable.dynamic.scale.vm to true 2. Create a custom Compute Offerings A 3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M 4. Create another custom Compute Offerings B 5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M (ensure 4 times over previous memory size), then you will encounter scaling failed 6. Stop VM instance , you will never startup again EXPECTED BEHAVIOR ================== Succeed Startup VM instance ACTUAL BEHAVIOR ================== Fail to start instance RCA: The ROLLBACK does not take care of restoring old service offering details. In case failure we are removing the new service offering details but restoring old service offering details is missing. Before Fix: `user_vm_details before upgrade. mysql> select * from user_vm_details where vm_id =9; +-----+-------+------------------------------------+-------------------------------------------------+---------+ | id | vm_id | name | value | display | +-----+-------+------------------------------------+-------------------------------------------------+---------+ | 118 | 9 | platform | viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 | | 119 | 9 | hypervisortoolsversion | xenserver56 | 1 | | 120 | 9 | Message.ReservedCapacityFreed.Flag | false | 1 | | 121 | 9 | cpuNumber | 1 | 1 | | 122 | 9 | cpuSpeed | 1000 | 1 | | 123 | 9 | memory | 256 | 1 | +-----+-------+------------------------------------+-------------------------------------------------+---------+ 6 rows in set (0.00 sec) user_vm_details after unsuccessful upgrade. mysql> select * from user_vm_details where vm_id =9; +-----+-------+------------------------------------+-------------------------------------------------+---------+ | id | vm_id | name | value | display | +-----+-------+------------------------------------+-------------------------------------------------+---------+ | 133 | 9 | platform | viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 | | 134 | 9 | hypervisortoolsversion | xenserver56 | 1 | | 135 | 9 | Message.ReservedCapacityFreed.Flag | false | 1 | +-----+-------+------------------------------------+-------------------------------------------------+---------+ 3 rows in set (0.00 sec)` After fix: ` mysql> select * from user_vm_details where vm_id =9; +-----+-------+------------------------------------+-------------------------------------------------+---------+ | id | vm_id | name | value | display | +-----+-------+------------------------------------+-------------------------------------------------+---------+ | 166 | 9 | cpuNumber | 1 | 1 | | 167 | 9 | platform | viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 | | 168 | 9 | cpuSpeed | 1000 | 1 | | 169 | 9 | Message.ReservedCapacityFreed.Flag | false | 1 | | 170 | 9 | memory | 256 | 1 | | 171 | 9 | hypervisortoolsversion | xenserver56 | 1 | +-----+-------+------------------------------------+-------------------------------------------------+---------+ 6 rows in set (0.00 sec) ` * pr/1796: CLOUDSTACK-9626: Instance fails to start after unsuccesful compute offering upgrade. Signed-off-by: Rohit Yadav > Instance fails to start after unsuccesful compute offering upgrade. > ------------------------------------------------------------------- > > Key: CLOUDSTACK-9626 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9626 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: Management Server > Affects Versions: 4.8.0 > Reporter: Sudhansu Sahu > Assignee: Sudhansu Sahu > > ISSUE > ============ > Instance fails to start after unsuccesful compute offering upgrade. > > TROUBLESHOOTING > ================== > We observed VM instance get compute values "cpuNumber","cpuSpeed","memory" removed from table "user_vm_details", which cause instance fail to startup next time on XenServer > mysql> select * from user_vm_details where vm_id=10; > +-----+-------+------------------------------------+-------------------------------------------------+---------+ > | id | vm_id | name | value | display | > +-----+-------+------------------------------------+-------------------------------------------------+---------+ > | 218 | 10 | platform | viridian:true;acpi:1;apic:true;pae:true;nx:true | 1 | > | 219 | 10 | hypervisortoolsversion | xenserver56 | 1 | > | 220 | 10 | Message.ReservedCapacityFreed.Flag | true | 1 | > +-----+-------+------------------------------------+-------------------------------------------------+---------+ > 3 rows in set (0.00 sec) > > Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.ScaleVMCmd > java.lang.NullPointerException > at com.cloud.vm.UserVmManagerImpl.upgradeStoppedVirtualMachine(UserVmManagerImpl.java:953) > at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1331) > at com.cloud.vm.UserVmManagerImpl.upgradeVirtualMachine(UserVmManagerImpl.java:1271) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50) > at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy169.upgradeVirtualMachine(Unknown Source) > at org.apache.cloudstack.api.command.user.vm.ScaleVMCmd.execute(ScaleVMCmd.java:127) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:167) > at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:97) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:543) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:50) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:47) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:500) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Select by Id SQL: SELECT user_vm_details.id, user_vm_details.vm_id, user_vm_details.name, user_vm_details.value, user_vm_details.display FROM user_vm_details WHERE user_vm_details.id = ? > 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Table References: user_vm_details > 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Insert SQLs: > 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) INSERT INTO user_vm_details (user_vm_details.vm_id, user_vm_details.name, user_vm_details.value, user_vm_details.display) VALUES (?, ?, ?, ?) > 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) Delete SQLs > 2015-02-09 15:23:46,578 TRACE [c.c.u.d.GenericDaoBase] (main:null) DELETE FROM user_vm_details WHERE user_vm_details.id= ? > REPRO STEPS > ================== > 1. Set global setting enable.dynamic.scale.vm to true > 2. Create a custom Compute Offerings A > 3. Create a VM instance apply A, ie. cpuNumber=1,cpuSpeed=1000,memory=512M > 4. Create another custom Compute Offerings B > 5. Change service offering to B, ie. cpuNumber=2,cpuSpeed=2000,memory=4096M (ensure 4 times over previous memory size), then you will encounter scaling failed > 6. Stop VM instance , you will never startup again > EXPECTED BEHAVIOR > ================== > Succeed Startup VM instance > > ACTUAL BEHAVIOR > ================== > Fail to start instance -- This message was sent by Atlassian JIRA (v6.3.4#6332)