Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6186D10B7E for ; Fri, 20 Dec 2013 00:19:07 +0000 (UTC) Received: (qmail 51253 invoked by uid 500); 20 Dec 2013 00:19:07 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 51183 invoked by uid 500); 20 Dec 2013 00:19:07 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 51175 invoked by uid 500); 20 Dec 2013 00:19:07 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 51172 invoked by uid 99); 20 Dec 2013 00:19:07 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Dec 2013 00:19:07 +0000 Date: Fri, 20 Dec 2013 00:19:07 +0000 (UTC) From: "edison su (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-5430) KVM - Primary store down - Not abel to start Vms/take snapshots after the primary store is brought down and brough back up again. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-5430?page=3Dcom.atl= assian.jira.plugin.system.issuetabpanels:all-tabpanel ] edison su updated CLOUDSTACK-5430: ---------------------------------- Assignee: Mike Tutkowski (was: edison su) > KVM - Primary store down - Not abel to start Vms/take snapshots after the= primary store is brought down and brough back up again. > -------------------------------------------------------------------------= -------------------------------------------------------- > > Key: CLOUDSTACK-5430 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-543= 0 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Management Server > Affects Versions: 4.3.0 > Environment: Build from 4.3 > Reporter: Sangeetha Hariharan > Assignee: Mike Tutkowski > Priority: Critical > Fix For: 4.3.0 > > Attachments: psdown.rar > > > KVM - Primary store down - Not abel to start Vms/take snapshots after the= primary store is brought down and brough back up again. > Set up: > Advanced zone with KVM (RHEL 6.3) hosts. > Steps to reproduce the problem: > 1. Deploy few Vms in each of the hosts with 10 GB ROOT volume size , so w= e start with 10 Vms. > 2. Create snaposhot for ROOT volumes. > 3. When snapshot is still in progress , Make the primary storage unavaila= ble for 10 mts. > This results in the KVM hosts to reboot. > But reboot of KVM host is not successful.It is stuck at trying to unmount= nfs mount points. This is tracked in CLOUDSTACK-5429. > Stop and start KM hosts manually to workaround this problem. > At this point all the Vms are marked as "Stopped" state in CloudStack. > 4. Now make the primary store available. > 5. Attempt to start the VM. > It fails to start with the following exception: > 2013-12-09 20:35:55,891 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:n= ull) Seq 2-1983250480: Processing: { Ans > : , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agen= t.api.Answer":{"result":false,"details":" > java.lang.NullPointerException\n\tat com.cloud.hypervisor.kvm.resource.Li= bvirtComputingResource.execute(LibvirtCom > putingResource.java:2488)\n\tat com.cloud.hypervisor.kvm.resource.Libvirt= ComputingResource.executeRequest(LibvirtC > omputingResource.java:1260)\n\tat com.cloud.agent.Agent.processRequest(Ag= ent.java:498)\n\tat com.cloud.agent.Agent > $AgentRequestHandler.doTask(Agent.java:806)\n\tat com.cloud.utils.nio.Tas= k.run(Task.java:83)\n\tat java.util.concu > rrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat ja= va.util.concurrent.ThreadPoolExecutor$Wor > ker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.ja= va:679)\n","wait":0}}] } > 2013-12-09 20:35:55,891 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-f0d= 35c47) Seq 2-1983250480: Received: { Ans > : , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10, { Answer } } > 2013-12-09 20:35:56,939 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-35= adede4) =3D=3D=3DSTART=3D=3D=3D 10.216.50.147 -- GET > command=3DqueryAsyncJobResult&jobId=3D489806e9-96f9-4940-9ea0-6bd9516aa= bb0&response=3Djson&sessionkey=3DqRSeXYRCfc1PSAXc > omRT8ue1f%2BE%3D&_=3D1386639381768 > 2013-12-09 20:35:56,953 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-35= adede4 ctx-065180b8) =3D=3D=3DEND=3D=3D=3D 10.216.50 > .147 -- GET command=3DqueryAsyncJobResult&jobId=3D489806e9-96f9-4940-9ea= 0-6bd9516aabb0&response=3Djson&sessionkey=3DqRSeX > YRCfc1PSAXcomRT8ue1f%2BE%3D&_=3D1386639381768 > 2013-12-09 20:35:59,322 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:= null) Seq 1-539557989: Processing: { Ans > : , MgmtId: 82324189320212, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agen= t.api.Answer":{"result":false,"details":" > java.lang.NullPointerException\n\tat com.cloud.hypervisor.kvm.storage.KVM= StoragePoolManager.disconnectPhysicalDisk > sViaVmSpec(KVMStoragePoolManager.java:181)\n\tat com.cloud.hypervisor.kvm= .resource.LibvirtComputingResource.execut > e(LibvirtComputingResource.java:3672)\n\tat com.cloud.hypervisor.kvm.reso= urce.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:= 1282)\n\tat com.cloud.agent.Agent.processRequest(Agent.java:498)\n\tat com.= cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:806)\n\tat com.clou= d.utils.nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExe= cutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.Th= readPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Th= read.run(Thread.java:679)\n","wait":0}},{"com.cloud.agent.api.Answer":{"res= ult":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.a= gent.api.Answer":{"result":false,"details":"Stopped by previous failure","w= ait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped b= y previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":fals= e,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.= Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}]= } > 2013-12-09 20:35:59,322 DEBUG [c.c.a.t.Request] (Job-Executor-26:ctx-0382= e21d ctx-d8f9d323) Seq 1-539557989: Received: { Ans: , MgmtId: 82324189320= 212, via: 1, Ver: v1, Flags: 10, { Answer, Answer, Answer, Answer, Answer, = Answer } } > 6. Attempting to take snapshots also fails with following exception: > 2013-12-09 20:54:10,509 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:= null) Seq 2-1983250525: Processing: { An > s: , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10, [{"org.apache.cl= oudstack.storage.command.CreateObjectAnsw > er":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeExce= ption: java.lang.NullPointerException","w > ait":0}}] } > 2013-12-09 20:54:10,509 DEBUG [c.c.a.t.Request] (Job-Executor-34:ctx-eb23= 7191 ctx-20bb478f) Seq 2-1983250525: Rece > ived: { Ans: , MgmtId: 82324189320212, via: 2, Ver: v1, Flags: 10, { Cre= ateObjectAnswer } } > 2013-12-09 20:54:10,509 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Job-Execut= or-34:ctx-eb237191 ctx-20bb478f) create s > napshot TestVM-tiny-host-0ps-0-4_ROOT-49_20131210014410 failed: com.cloud= .utils.exception.CloudRuntimeException: j > ava.lang.NullPointerException > 2013-12-09 20:54:10,519 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy] (Job-= Executor-34:ctx-eb237191 ctx-20bb478f) Fa > iled to take snapshot: com.cloud.utils.exception.CloudRuntimeException: j= ava.lang.NullPointerException > 2013-12-09 20:54:10,536 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor= -34:ctx-eb237191 ctx-20bb478f) Failed to > create snapshot > com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exceptio= n.CloudRuntimeException: java.lang.NullPo > interException > at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrate= gy.takeSnapshot(XenserverSnapshotStrategy > .java:281) > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(Sn= apshotManagerImpl.java:951) > at sun.reflect.GeneratedMethodAccessor230.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingR= eflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.i= nvokeJoinpoint(ReflectiveMethodInvocation > .java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:15 > 0) > at org.springframework.aop.interceptor.ExposeInvocationIntercepto= r.invoke(ExposeInvocationInterceptor.java > :91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:17 > 2) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Jd= kDynamicAopProxy.java:204) > at $Proxy161.takeSnapshot(Unknown Source) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSna= pshot(VolumeServiceImpl.java:1341) > at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiS= erviceImpl.java:1461) > at sun.reflect.GeneratedMethodAccessor229.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth= odAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingR= eflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation.i= nvokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:150) > at org.springframework.aop.interceptor.ExposeInvocationIntercepto= r.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation.p= roceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Jd= kDynamicAopProxy.java:204) > at $Proxy233.takeSnapshot(Unknown Source) > at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshot= Cmd.execute(CreateSnapshotCmd.java:181) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJo= bDispatcher.java:109) > at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatche= r.java:66) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatch= er.java:63) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$= 5.runInContext(AsyncJobManagerImpl.java:520) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1= .run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedConte= xt.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.r= un(ManagedContextRunnable.java:46) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.= java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:= 334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2013-12-09 20:54:10,544 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor= -34:ctx-eb237191 ctx-20bb478f) Take snapshot: 49 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapsho= t -- This message was sent by Atlassian JIRA (v6.1.4#6159)