Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C82F210BFB for ; Thu, 9 Jan 2014 00:30:51 +0000 (UTC) Received: (qmail 32494 invoked by uid 500); 9 Jan 2014 00:30:51 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 32468 invoked by uid 500); 9 Jan 2014 00:30:51 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 32434 invoked by uid 500); 9 Jan 2014 00:30:51 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 32421 invoked by uid 99); 9 Jan 2014 00:30:51 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jan 2014 00:30:51 +0000 Date: Thu, 9 Jan 2014 00:30:51 +0000 (UTC) From: "Sheng Yang (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (CLOUDSTACK-4540) [Automation] Parallel deployment - Vmware - When deploying 30 parallel Vms , 16 Vms fails to get deployed due to "VmDataCommand failed due to Exception: java.lang.Exception Message: Timed out in waiting SSH execution result" MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sheng Yang resolved CLOUDSTACK-4540. ------------------------------------ Resolution: Done This problem already have a proven workaround, but we cannot make it as default value due to other limitation. Make the improvement an separate issue and close this one. https://issues.apache.org/jira/browse/CLOUDSTACK-5842 > [Automation] Parallel deployment - Vmware - When deploying 30 parallel Vms , 16 Vms fails to get deployed due to "VmDataCommand failed due to Exception: java.lang.Exception Message: Timed out in waiting SSH execution result" > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-4540 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4540 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: Management Server > Affects Versions: 4.2.0 > Environment: Build from 4.2-forward. > Reporter: Sangeetha Hariharan > Assignee: Sheng Yang > Priority: Blocker > Fix For: 4.3.0 > > Attachments: management-server.log > > > Parallel deployment - Vmware - When deploying 30 parallel Vms , 16 Vms fails to get deployed due to "VmDataCommand failed due to Exception: java.lang.Exception > Message: Timed out in waiting SSH execution result" > Set up - Advanced zone with 1 Vmware 5.0.0 Esxi host. > Deploy 30 Vms in parallel. > 16 out of 30 vms deployed in parallel , failed due to "VmDataCommand failed due to Exception: java.lang.Exception > Message: Timed out in waiting SSH execution result" > Following exception seen in Management server logs: > 2013-08-28 10:26:58,939 ERROR [vmware.resource.VmwareResource] (DirectAgent-21:10.223.58.66) VmDataCommand failed due to Exception: java.lang.Exception > Message: Timed out in waiting SSH execution result > java.lang.Exception: Timed out in waiting SSH execution result > at com.cloud.utils.ssh.SshHelper.sshExecute(SshHelper.java:166) > at com.cloud.utils.ssh.SshHelper.sshExecute(SshHelper.java:37) > at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2470) > at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:441) > at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-08-28 10:26:58,940 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-21:null) Seq 1-170983503: Response Received: > 2013-08-28 10:26:58,941 DEBUG [agent.transport.Request] (DirectAgent-21:null) Seq 1-170983503: Processing: { Ans: , MgmtId: 7083743249448, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"VmDataCommand failed due to Exception: java.lang.Exception\nMessage: Timed out in waiting SSH execution result\n","wait":0}}] } > 2013-08-28 10:26:58,941 DEBUG [agent.transport.Request] (Job-Executor-29:job-398 = [ b3a34f25-37b2-4f33-b183-c0ea348d7af9 ]) Seq 1-170983503: Received: { Ans: , MgmtId: 7083743249448, via: 1, Ver: v1, Flags: 10, { Answer, Answer } } > 2013-08-28 10:26:58,979 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-29:job-398 = [ b3a34f25-37b2-4f33-b183-c0ea348d7af9 ]) Unable to contact resource. > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply userdata and password entry on router > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3808) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyUserData(VirtualNetworkApplianceManagerImpl.java:2993) > at com.cloud.network.element.VirtualRouterElement.addPasswordAndUserdata(VirtualRouterElement.java:926) > at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2076) > at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2191) > at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2127) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) > at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227) > at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2966) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2952) > at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) -- This message was sent by Atlassian JIRA (v6.1.5#6160)