Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7FCF817DAD for ; Sat, 8 Nov 2014 00:36:35 +0000 (UTC) Received: (qmail 62457 invoked by uid 500); 8 Nov 2014 00:36:35 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 62432 invoked by uid 500); 8 Nov 2014 00:36:35 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 62421 invoked by uid 500); 8 Nov 2014 00:36:35 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 62418 invoked by uid 99); 8 Nov 2014 00:36:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Nov 2014 00:36:35 +0000 Date: Sat, 8 Nov 2014 00:36:35 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-7864) CPVM continues to be in "STOPPED' state after failure to start because of a management server restart. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-7864?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D14203030#comment-14203030 ]=20 ASF subversion and git services commented on CLOUDSTACK-7864: ------------------------------------------------------------- Commit a275bd7ca4191f19defcfe106229a1bdc33c6d99 in cloudstack's branch refs= /heads/master from [~minchen07] [ https://git-wip-us.apache.org/repos/asf?p=3Dcloudstack.git;h=3Da275bd7 ] CLOUDSTACK-7864: CPVM continues to be in Stopped state after a failure to s= tart because of a management server restart. > CPVM continues to be in "STOPPED' state after failure to start because of= a management server restart. > -------------------------------------------------------------------------= ----------------------------- > > Key: CLOUDSTACK-7864 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-786= 4 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the defa= ult.)=20 > Components: Management Server > Affects Versions: 4.5.0 > Reporter: Min Chen > Assignee: Min Chen > Priority: Critical > Fix For: 4.5.0 > > > CPVM continues to be in "STOPPED' state after failure to start because of= a management server restart. > Setup: > Advanced zone set up with KVM hosts. > Steps that led to the problem: > As soon as the zone was set up and enabled , management server was restar= ted. > I see that the CPVM failed to come up because there were no hosts at the = time it was attempting to start. > After this , even manually trying to start the CPVM does not succeed and = the jobs are hanging. > 2014-11-04 21:29:02,613 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Execu= tor-1:ctx-5e9c6a9d job-3/job-16) Unable to complete AsyncJobVO > {id:16, userId: 1, accountId: 1, instanceType: null, instanceId: null, cm= d: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU= 3RhcnR9cMGsvxz73gIACkoABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBs= b3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc= 7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeX= NpY2FsTmV0d29ya0lkcQB-AAJMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhb= XN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZt= LlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJ= OYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAXQAGVZpcnR1YWxNYWNoaW5lTWFuYW= dlckltcGwAAAAAAAAAAHBwcHBwcHBwcA, cmdVersion: 0, status: FAILED, processSta= tus: 0, resultCode: 530, result: job cancelled because of management server= restart or shutdown, initMsid: 235397185403991, completeMsid: null, lastUp= dated: null, lastPolled: null, created: Tue Nov 04 21:28:01 UTC 2014} > , job origin:3 > com.cloud.exception.InsufficientServerCapacityException: Unable to create= a deployment for VM[ConsoleProxy|v-1-VM]Scope=3Dinterface com.cloud.dc.Dat= aCenter; id=3D1 > at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachine= ManagerImpl.java:932) > at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachine= ManagerImpl.java:5012) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j= ava:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerPro= xy.java:107) > at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineM= anagerImpl.java:5156) > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:101) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInC= ontext(AsyncJobManagerImpl.java:540) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man= agedContextRunnable.java:50) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal= l(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW= ithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi= thContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag= edContextRunnable.java:47) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(As= yncJobManagerImpl.java:497) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471= ) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:603) > at java.lang.Thread.run(Thread.java:679) > 2014-11-04 21:29:02,620 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job= -Executor-1:ctx-5e9c6a9d job-3/job-16) Complete async job-16, jobStatus: FA= ILED, resultCode: 0, result: rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp= 5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhb= mcuVGhyb3dhYmxl1cY1Jzl3uMsDAANMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wA= DWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZ= hL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7eHBxAH4ABnQAVUpvYiBmYWlsZWQgZHVlIHRvIGV4Y2= VwdGlvbiBVbmFibGUgdG8gY3JlYXRlIGEgZGVwbG95bWVudCBmb3IgVk1bQ29uc29sZVByb3h5f= HYtMS1WTV11cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAA= AA5zcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmV= yTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABEwACGZpbGVOYW1lcQB-AARMAAptZXRob2ROYW1lcQB-AA= R4cAAAAHF0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc= 3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ACgAAAhx0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2su= ZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmF= nZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQB-AAoAAAAydAA-b3JnLmFwYWNoZS5jbG91ZH= N0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ= 29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AAoAAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0= YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXV= sdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfgAKAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdW= RzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AGXQAD= 2NhbGxXaXRoQ29udGV4dHNxAH4ACgAAADVxAH4AHHEAfgAZdAAOcnVuV2l0aENvbnRleHRzcQB-= AAoAAAAvdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29= udGV4dFJ1bm5hYmxlcQB-ABVxAH4AFnNxAH4ACgAAAfFxAH4AEHEAfgARcQB-ABZzcQB-AAoAAA= HXdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4Z= WN1dG9ycy5qYXZhcQB-ABpzcQB-AAoAAAFOdAAkamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJl= VGFzayRTeW5jdAAPRnV0dXJlVGFzay5qYXZhdAAIaW5uZXJSdW5zcQB-AAoAAACmdAAfamF2YS5= 1dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3EAfgAocQB-ABZzcQB-AAoAAARWdAAnamF2YS51dG= lsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphd= mF0AAlydW5Xb3JrZXJzcQB-AAoAAAJbdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9v= bEV4ZWN1dG9yJFdvcmtlcnEAfgAucQB-ABZzcQB-AAoAAAKndAAQamF2YS5sYW5nLlRocmVhZHQ= AC1RocmVhZC5qYXZhcQB-ABZ4 > 2014-11-04 21:29:02,636 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job= -Executor-1:ctx-5e9c6a9d job-3/job-16) job-16 is already completed. > 2014-11-04 21:29:02,636 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job= -Executor-1:ctx-5e9c6a9d job-3/job-16) Done executing com.cloud.vm.VmWorkSt= art for job-16 > 2014-11-04 21:29:02,650 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] (Work-Jo= b-Executor-1:ctx-5e9c6a9d job-3/job-16) Queue (queue id, sync type, sync id= ) - (1,VmWorkJobQueue, 1) is reaching concurrency limit 1 > 2014-11-04 21:29:02,651 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] (Work-Jo= b-Executor-1:ctx-5e9c6a9d job-3/job-16) There is a pending process in sync = queue(id: 1) > 2014-11-04 21:29:02,651 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Exec= utor-1:ctx-5e9c6a9d job-3/job-16) Remove job-16 from job monitoring -- This message was sent by Atlassian JIRA (v6.3.4#6332)