Return-Path: X-Original-To: apmail-cloudstack-dev-archive@www.apache.org Delivered-To: apmail-cloudstack-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 45DE6171D7 for ; Sat, 14 Mar 2015 16:06:40 +0000 (UTC) Received: (qmail 79876 invoked by uid 500); 14 Mar 2015 16:06:39 -0000 Delivered-To: apmail-cloudstack-dev-archive@cloudstack.apache.org Received: (qmail 79826 invoked by uid 500); 14 Mar 2015 16:06:39 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 79814 invoked by uid 99); 14 Mar 2015 16:06:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Mar 2015 16:06:39 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shadowsor@gmail.com designates 209.85.215.52 as permitted sender) Received: from [209.85.215.52] (HELO mail-la0-f52.google.com) (209.85.215.52) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Mar 2015 16:06:35 +0000 Received: by ladw1 with SMTP id w1so10755840lad.0 for ; Sat, 14 Mar 2015 09:04:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=NE2PTZ9lufbO7C8jSHZDRxFnaKcpJs+QX5/xaHh3Ms4=; b=i5loNTZS9TdaIFtBXgiPiPsy7FDXlb0IWRLWeNzKkw3/OwSrLwkd+7fiOmihE2+Ewl bSpi5NxXlP426i4DH5/OMya40j1vlq4u4JhCzQjTHbLdE2HBf/o0r16VoTP2+VcYecK3 Od/x7ccoa9gukFCbQDJgMUpxm+K3NwUS3hwj0UYz9Lk3c/t2YwyjdYlOXHxsaFMBU8Jg tD6SQrZJzekRQTiSheq7MgJss9BwARJprent7jZXlUMP6duAHQVCgncAEo9QHUi/5oYA VEj0rpbowp6nhtCCxErD4a/iIGqbkfEkxrC9hiBtGCYsmaQDLMiEtqeIn18cAnK0kExB 53Mw== MIME-Version: 1.0 X-Received: by 10.152.10.66 with SMTP id g2mr47699806lab.44.1426349084230; Sat, 14 Mar 2015 09:04:44 -0700 (PDT) Received: by 10.25.139.8 with HTTP; Sat, 14 Mar 2015 09:04:44 -0700 (PDT) In-Reply-To: References: <54FF20BE.6000902@gmail.com> <54FF65F4.2030103@exoscale.ch> <5500DEFD.8040508@gmail.com> Date: Sat, 14 Mar 2015 09:04:44 -0700 Message-ID: Subject: Re: [HELP] VR sometimes fail to start From: Marcus To: "dev@cloudstack.apache.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Yeah, I don't remember all of the discussion around the minimum version and why, but each libvirt and qemu has such different behavior (not all good), that we decided to target a minimum regardless of what was supplied by the distros. It's just too hard to support them all with their differing behavior. Open stack ran into the same issues. On Sat, Mar 14, 2015 at 9:02 AM, Andrija Panic wr= ote: > Marcus - QEMU is 0.12.x (CentOS 6.4) > > but libvirt is compiled 1.2.3 > > On 14 March 2015 at 16:58, Marcus wrote: > >> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and >> Qemu 1.0+ per the install docs. >> >> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic >> wrote: >> > Tried the proposed solution - but it still don't work - BTW my qemu is >> > 0.12.x - from CentOS 6.4... >> > >> > When I manually start the script with all parameters, it simply hangs,= no >> > output at all, until ctrl + c >> > >> > BTW this is with now upgraded ACS to 4.3.2... >> > >> > :( >> > >> > On 12 March 2015 at 09:24, Andrija Panic >> wrote: >> > >> >> Hi Marcus, will do - I'm just unconfortable changing live things, but >> that >> >> is my problem. Will let you know of the output. >> >> >> >> Thanks >> >> >> >> On 12 March 2015 at 01:34, ilya musayev >> >> wrote: >> >> >> >>> Andrija, >> >>> >> >>> Have you tried the solution Loic mentioned? Its seems like an easy o= ne >> to >> >>> try.. >> >>> >> >>> Thanks >> >>> ilya >> >>> >> >>> >> >>> On 3/11/15 3:54 AM, Andrija Panic wrote: >> >>> >> >>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the >> problem >> >>>> go >> >>>> away... >> >>>> >> >>>> THanks guys >> >>>> >> >>>> On 10 March 2015 at 22:45, Loic Lambiel >> >>>> wrote: >> >>>> >> >>>> Hi, >> >>>>> >> >>>>> Maybe the same issue we faced here: https://github.com/exoscale/ >> >>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd >> >>>>> >> >>>>> The patch has not been yet submitted upstream. I'll do it once it = is >> >>>>> running on our production. >> >>>>> >> >>>>> Anyway, you may easily test it by performing the small change >> directly >> >>>>> into the script. >> >>>>> >> >>>>> Hope that helps. >> >>>>> >> >>>>> Regards, >> >>>>> >> >>>>> Loic Lambiel >> >>>>> >> >>>>> >> >>>>> On 10/03/15 19:40, Andrija Panic wrote: >> >>>>> >> >>>>> Hi guys, >> >>>>>> >> >>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6) >> >>>>>> Libvirt manually compiled 1.2.3 with RBD support. >> >>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD >> >>>>>> support. >> >>>>>> >> >>>>>> And here is the output, with debug mode, from Agent. >> >>>>>> >> >>>>>> At the begining I see agent deleting routes for 169.254.x.x >> network...: >> >>>>>> >> >>>>>> Basically, It seems like the >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> is >> >>>>>> failing, not being able to connect to socket... >> >>>>>> And I could not ping VR (169.254.2.99)... >> >>>>>> >> >>>>>> LOG: >> >>>>>> >> >>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-4:) Processing command: >> >>>>>> com.cloud.agent.api.GetVmStatsCommand >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , >> MgmtId: >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111, >> >>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id": >> >>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1," >> >>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam": >> >>>>>> 268435456,"arch":"x86_64","os":"Debian >> >>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=3D10.0.0.0/8 >> >>>>>> domain=3Dcs2cloud.internal dns1=3D8.8.8.8 dns2=3D template=3DdomP >> >>>>>> name=3Dr-2191-VM >> >>>>>> eth0ip=3D169.254.2.99 eth0mask=3D255.255.0.0 type=3Dvpcrouter >> >>>>>> disable_rp_filter=3Dtrue","rebootOnCrash":false," >> >>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm": >> >>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{" >> >>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"}," >> >>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{" >> >>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{" >> >>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93"," >> >>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack. >> >>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d- >> >>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":" >> >>>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD:/= / >> >>>>>> cephmon.domaina.net/cloudstack/?ROLE=3DPrimary& >> >>>>>> STOREUUID=3D5b93422e-1a66-353d-88a8-2203f79b1dc6 >> >>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003- >> >>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r- >> >>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857," >> >>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":" >> >>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_ >> >>>>>> details":{"managed":"false","storagePort":"6789","storageHost":" >> >>>>>> cephmon.domaina.net >> >>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0," >> >>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":" >> >>>>>> d27a7aad-2dcc-45c7-bf30- >> >>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0"," >> >>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63"," >> >>>>>> broadcastType":"LinkLocal","type":"Control"," >> >>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180"," >> >>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent. >> >>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port": >> >>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait": >> >>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{" >> >>>>>> router.name >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{" >> >>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{" >> >>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid" >> >>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146. >> >>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1"," >> >>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":" >> >>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":" >> >>>>>> >> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"}," >> >>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait": >> >>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{" >> >>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62"," >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP": >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5. >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":" >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public"," >> >>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest. >> >>>>>> ip":"5.226.146.62","zone.network.type":"Advanced"," >> >>>>>> router.name >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{" >> >>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{" >> >>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62"," >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP": >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5. >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":" >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public"," >> >>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{" >> >>>>>> zone.network.type":"Advanced"," >> >>>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{= }] >> } >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-1:) Processing command: >> >>>>>> com.cloud.agent.api.StartCommand >> >>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-5:) Processing command: >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand >> >>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper= ] >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Deb= ian >> >>>>>> GNU/Linux 7(64-bit) >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-2:) Processing command: >> >>>>>> com.cloud.agent.api.NetworkUsageCommand >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Executing: >> >>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh >> >>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g >> >>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Execution is successful. >> >>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId: >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 10, >> >>>>>> >> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM"," >> >>>>>> >> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":""," >> >>>>>> wait":0}}] >> >>>>>> } >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver] >> >>>>>> (agentRequest-Handler-1:) nic=3D[Nic:Control-169.254.2.99-null] >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script] >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep >> >>>>>> 169.254.0.0/16 >> >>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script] >> >>>>>> (agentRequest-Handler-1:) Execution is successful. >> >>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script] >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del >> >>>>>> 169.254.0.0/16 >> >>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script] >> >>>>>> (agentRequest-Handler-1:) Execution is successful. >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper= ] >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os: Deb= ian >> >>>>>> GNU/Linux 7(64-bit) >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) starting r-2191-VM: >> >>>>>> r-2191-VM >> >>>>>> 485587f7-4a04-4d77-bc36-2367f2adf77e >> >>>>>> Debian GNU/Linux 7(64-bit) >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> /usr/libexec/qemu-kvm >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> > >>>>>> passwd=3D'e9e24eb0b1ff8ec2'/> >> >>>>>> >> >>>>>> >> >>>>>> > >>>>>> name=3D'cloudstack/d042e003-5278-461d-8a35-3b1261164f93'> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>>
>> >>>>>> >> >>>>>> >> >>>>>> 262144 >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> 1 >> >>>>>> >> >>>>>> hvm >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> 166 >> >>>>>> >> >>>>>> restart >> >>>>>> destroy >> >>>>>> destroy >> >>>>>> >> >>>>>> >> >>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Executing: >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> -n >> >>>>>> r-2191-VM -p %vpccidr=3D >> >>>>>> 10.0.0.0/8%domain=3Dcs2cloud.internal%dns1=3D8.8.8.8%dns2=3D% >> >>>>>> template=3DdomP%name=3Dr-2191-VM%eth0ip=3D169.254.2.99%eth0mask= =3D >> >>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>> >> >>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Execution is successful. >> >>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> >>>>>> (Thread-10:) >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in >> >>>>>> libvirt, >> >>>>>> continuing >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> >>>>>> (Thread-10:) >> >>>>>> Executing: >> >>>>>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh >> >>>>>> -i >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180 >> >>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> >>>>>> (Thread-10:) >> >>>>>> Execution is successful. >> >>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] >> (UgentTask-5:) >> >>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, >> Flags: >> >>>>>> 11, >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_ >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":" >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs >> >>>>>> 18.domaina.net >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true," >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}] >> >>>>>> } >> >>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (Agent-Handler-3:) >> >>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via= : >> 788, >> >>>>>> Ver: v1, Flags: 100010, >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{" >> >>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0= }}] >> >>>>>> } >> >>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Unable to logon to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Executing: >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> -n >> >>>>>> r-2191-VM -p %vpccidr=3D >> >>>>>> 10.0.0.0/8%domain=3Dcs2cloud.internal%dns1=3D8.8.8.8%dns2=3D% >> >>>>>> template=3DdomP%name=3Dr-2191-VM%eth0ip=3D169.254.2.99%eth0mask= =3D >> >>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>> >> >>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Execution is successful. >> >>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-3:) Processing command: >> >>>>>> com.cloud.agent.api.GetHostStatsCommand >> >>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=3D$(top -b= -n >> >>>>>> 1|grep >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle >> >>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-3:) Execution is successful. >> >>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c >> freeMem=3D$(free|grep >> >>>>>> cache:|awk '{print $4}');echo $freeMem >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-3:) Execution is successful. >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|= awk >> >>>>>> '{print $2}' >> >>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-3:) Execution is successful. >> >>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Unable to logon to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Executing: >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> -n >> >>>>>> r-2191-VM -p %vpccidr=3D >> >>>>>> 10.0.0.0/8%domain=3Dcs2cloud.internal%dns1=3D8.8.8.8%dns2=3D% >> >>>>>> template=3DdomP%name=3Dr-2191-VM%eth0ip=3D169.254.2.99%eth0mask= =3D >> >>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>> >> >>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Execution is successful. >> >>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-4:) Processing command: >> >>>>>> com.cloud.agent.api.GetVmStatsCommand >> >>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Unable to logon to 169.254.2.99 >> >>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Executing: >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> -n >> >>>>>> r-2191-VM -p %vpccidr=3D >> >>>>>> 10.0.0.0/8%domain=3Dcs2cloud.internal%dns1=3D8.8.8.8%dns2=3D% >> >>>>>> template=3DdomP%name=3Dr-2191-VM%eth0ip=3D169.254.2.99%eth0mask= =3D >> >>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Execution is successful. >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-5:) Processing command: >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Unable to logon to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Executing: >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> -n >> >>>>>> r-2191-VM -p %vpccidr=3D >> >>>>>> 10.0.0.0/8%domain=3Dcs2cloud.internal%dns1=3D8.8.8.8%dns2=3D% >> >>>>>> template=3DdomP%name=3Dr-2191-VM%eth0ip=3D169.254.2.99%eth0mask= =3D >> >>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Execution is successful. >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> >>>>>> (Thread-11:) >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in >> >>>>>> libvirt, >> >>>>>> continuing >> >>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> >>>>>> (Thread-11:) >> >>>>>> Executing: >> >>>>>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh >> >>>>>> -i >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180 >> >>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> >>>>>> (Thread-11:) >> >>>>>> Execution is successful. >> >>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] >> (UgentTask-5:) >> >>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, >> Flags: >> >>>>>> 11, >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_ >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":" >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs >> >>>>>> 18.domaina.net >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true," >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}] >> >>>>>> } >> >>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (Agent-Handler-2:) >> >>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via= : >> 788, >> >>>>>> Ver: v1, Flags: 100010, >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{" >> >>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0= }}] >> >>>>>> } >> >>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Unable to logon to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Executing: >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> -n >> >>>>>> r-2191-VM -p %vpccidr=3D >> >>>>>> 10.0.0.0/8%domain=3Dcs2cloud.internal%dns1=3D8.8.8.8%dns2=3D% >> >>>>>> template=3DdomP%name=3Dr-2191-VM%eth0ip=3D169.254.2.99%eth0mask= =3D >> >>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>> >> >>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (Script-5:) Interrupting script. >> >>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) Timed out: >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>> hviasocket.pl >> >>>>>> -n >> >>>>>> r-2191-VM -p %vpccidr=3D >> >>>>>> 10.0.0.0/8%domain=3Dcs2cloud.internal%dns1=3D8.8.8.8%dns2=3D% >> >>>>>> template=3DdomP%name=3Dr-2191-VM%eth0ip=3D169.254.2.99%eth0mask= =3D >> >>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>> . Output is: >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-1:) passcmd failed:timeout >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-2:) Processing command: >> >>>>>> com.cloud.agent.api.GetHostStatsCommand >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=3D$(top -b= -n >> >>>>>> 1|grep >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle >> >>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Execution is successful. >> >>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c >> freeMem=3D$(free|grep >> >>>>>> cache:|awk '{print $4}');echo $freeMem >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Execution is successful. >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|= awk >> >>>>>> '{print $2}' >> >>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource. >> >>>>>> LibvirtComputingResource] >> >>>>>> (agentRequest-Handler-2:) Execution is successful. >> >>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Trying to connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Could not connect to 169.254.2.99 >> >>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent] >> >>>>>> (agentRequest-Handler-3:) Processing command: >> >>>>>> com.cloud.agent.api.GetVmStatsCommand >> >>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource] >> >>>>>> (agentRequest-Handler-1:) >> >>>>>> Unable to logon to 169.254.2.99 >> >>>>>> >> >>>>>> On 10 March 2015 at 17:50, ilya musayev < >> ilya.mailing.lists@gmail.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>> What kernel are you running? Please include Libvirt and Qemu >> version. >> >>>>>> >> >>>>>>> >> >>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote: >> >>>>>>> >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>>> we are having issues from time to time with starting VR - while >> >>>>>>>> creating >> >>>>>>>> VPC. >> >>>>>>>> >> >>>>>>>> Here is the few lines from MGMT log: >> >>>>>>>> >> >>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl] >> >>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc >> [VPC >> >>>>>>>> [621-VPC-delete] due to >> >>>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:7= 88] >> >>>>>>>> is >> >>>>>>>> unreachable: Host 788: Unable to start instance due to Unable t= o >> >>>>>>>> start >> >>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not >> >>>>>>>> retrying >> >>>>>>>> at >> >>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart( >> >>>>>>>> VirtualMachineManagerImpl.java:1072) >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> The interesting part is on the Agent itself: >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource. >> >>>>>>>> LibvirtComputingResource] >> >>>>>>>> (agentRequest-Handler-2:) Timed out: >> >>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >> >>>>>>>> hviasocket.pl >> >>>>>>>> -n >> >>>>>>>> r-2189-VM -p %vpccidr=3D >> >>>>>>>> 10.0.0.0/24%domain=3Dcs640cloud.internal%dns1=3D8.8.8.8%dns2=3D= % >> >>>>>>>> template=3DdomP%name=3Dr-2189-VM%eth0ip=3D169.254.2.134%eth0mas= k=3D >> >>>>>>>> 255.255.0.0%type=3Dvpcrouter%disable_rp_filter=3Dtrue >> >>>>>>>> . Output is: >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out of = 5 >> >>>>>>>> works. >> >>>>>>>> >> >>>>>>>> Any suggestion are really appreciated, since this is live syste= m >> :( >> >>>>>>>> ACS 4.3.0, KVM, CEPH >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> >> >> -- >> >> >> >> Andrija Pani=C4=87 >> >> >> > >> > >> > >> > -- >> > >> > Andrija Pani=C4=87 >> > > > > -- > > Andrija Pani=C4=87