cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus <shadow...@gmail.com>
Subject Re: [HELP] VR sometimes fail to start
Date Sat, 14 Mar 2015 15:59:16 GMT
People are mostly using upgraded 'virt' repos.

On Sat, Mar 14, 2015 at 8:58 AM, Marcus <shadowsor@gmail.com> wrote:
> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
> Qemu 1.0+ per the install docs.
>
> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <andrija.panic@gmail.com> wrote:
>> Tried the proposed solution - but it still don't work - BTW my qemu is
>> 0.12.x - from CentOS 6.4...
>>
>> When I manually start the script with all parameters, it simply hangs, no
>> output at all, until ctrl + c
>>
>> BTW this is with now upgraded ACS to 4.3.2...
>>
>> :(
>>
>> On 12 March 2015 at 09:24, Andrija Panic <andrija.panic@gmail.com> wrote:
>>
>>> Hi Marcus, will do - I'm just unconfortable changing live things, but that
>>> is my problem. Will let you know of the output.
>>>
>>> Thanks
>>>
>>> On 12 March 2015 at 01:34, ilya musayev <ilya.mailing.lists@gmail.com>
>>> wrote:
>>>
>>>> Andrija,
>>>>
>>>> Have you tried the solution Loic mentioned? Its seems like an easy one to
>>>> try..
>>>>
>>>> Thanks
>>>> ilya
>>>>
>>>>
>>>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>>>>
>>>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the problem
>>>>> go
>>>>> away...
>>>>>
>>>>> THanks guys
>>>>>
>>>>> On 10 March 2015 at 22:45, Loic Lambiel <loic.lambiel@exoscale.ch>
>>>>> wrote:
>>>>>
>>>>>  Hi,
>>>>>>
>>>>>> Maybe the same issue we faced here: https://github.com/exoscale/
>>>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>>>>>
>>>>>> The patch has not been yet submitted upstream. I'll do it once it
is
>>>>>> running on our production.
>>>>>>
>>>>>> Anyway, you may easily test it by performing the small change directly
>>>>>> into the script.
>>>>>>
>>>>>> Hope that helps.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Loic Lambiel
>>>>>>
>>>>>>
>>>>>> On 10/03/15 19:40, Andrija Panic wrote:
>>>>>>
>>>>>>  Hi guys,
>>>>>>>
>>>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>>>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>>>>>>> support.
>>>>>>>
>>>>>>> And here is the output, with debug mode, from Agent.
>>>>>>>
>>>>>>> At the begining I see agent deleting routes for 169.254.x.x network...:
>>>>>>>
>>>>>>> Basically, It seems like the
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> is
>>>>>>> failing, not being able to connect to socket...
>>>>>>> And I could not ping VR (169.254.2.99)...
>>>>>>>
>>>>>>> LOG:
>>>>>>>
>>>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-4:) Processing command:
>>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926:  { Cmd
, MgmtId:
>>>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>>>>>> 268435456,"arch":"x86_64","os":"Debian
>>>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>>>>>>> name=r-2191-VM
>>>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>>>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>>>>>> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD://
>>>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>>>>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>>>>>> cephmon.domaina.net
>>>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>>>>>>> d27a7aad-2dcc-45c7-bf30-
>>>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>>>>>> broadcastType":"LinkLocal","type":"Control","
>>>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>>>>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>>>>>> router.name
>>>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>>>>>> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>>>>>> router.name
>>>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>>>>>> zone.network.type":"Advanced","
>>>>>>> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}]
}
>>>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-1:) Processing command:
>>>>>>> com.cloud.agent.api.StartCommand
>>>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-5:) Processing command:
>>>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
Debian
>>>>>>> GNU/Linux 7(64-bit)
>>>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-2:) Processing command:
>>>>>>> com.cloud.agent.api.NetworkUsageCommand
>>>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-2:) Seq 788-1319632928:  { Ans: , MgmtId:
>>>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>>>>>> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>>>>>> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>>>>>> wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver]
>>>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route |
grep
>>>>>>> 169.254.0.0/16
>>>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>>>>>> 169.254.0.0/16
>>>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper]
>>>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
Debian
>>>>>>> GNU/Linux 7(64-bit)
>>>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'>
>>>>>>> <name>r-2191-VM</name>
>>>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>>>>>> <clock offset='utc'>
>>>>>>> </clock>
>>>>>>> <features>
>>>>>>> <pae/>
>>>>>>> <apic/>
>>>>>>> <acpi/>
>>>>>>> </features>
>>>>>>> <devices>
>>>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>>>>>> <interface type='bridge'>
>>>>>>> <source bridge='cloud0'/>
>>>>>>> <mac address='0e:00:a9:fe:02:63'/>
>>>>>>> <model type='virtio'/>
>>>>>>> </interface>
>>>>>>> <serial type='pty'>
>>>>>>> <target port='0'/>
>>>>>>> </serial>
>>>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>>>>>> passwd='e9e24eb0b1ff8ec2'/>
>>>>>>> <disk  device='disk' type='network'>
>>>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>>>> <source  protocol='rbd'
>>>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>>>>>> <host name='cephmon.domain.internal' port='6789'/>
>>>>>>> </source>
>>>>>>> <auth username='cloudstack'>
>>>>>>> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>>>>>> </auth>
>>>>>>> <target dev='vda' bus='virtio'/>
>>>>>>> </disk>
>>>>>>> <disk  device='cdrom' type='file'>
>>>>>>> <driver name='qemu' type='raw' cache='none' />
>>>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>>>>>> <target dev='hdc' bus='ide'/>
>>>>>>> </disk>
>>>>>>> <console type='pty'>
>>>>>>> <target port='0'/>
>>>>>>> </console>
>>>>>>> <input type='tablet' bus='usb'/>
>>>>>>> <channel type='unix'>
>>>>>>> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>>>>>>> <address type='virtio-serial'/>
>>>>>>> </channel>
>>>>>>> </devices>
>>>>>>> <memory>262144</memory>
>>>>>>> <devices>
>>>>>>> <memballoon model='none'/>
>>>>>>> </devices>
>>>>>>> <vcpu>1</vcpu>
>>>>>>> <os>
>>>>>>> <type  arch='x86_64' machine='pc'>hvm</type>
>>>>>>> <boot dev='cdrom'/>
>>>>>>> <boot dev='hd'/>
>>>>>>> </os>
>>>>>>> <cputune>
>>>>>>> <shares>166</shares>
>>>>>>> </cputune>
>>>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>>>>>> <on_poweroff>destroy</on_poweroff>
>>>>>>> <on_crash>destroy</on_crash>
>>>>>>> </domain>
>>>>>>>
>>>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-10:)
>>>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>>>> libvirt,
>>>>>>> continuing
>>>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-10:)
>>>>>>> Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>>>> -i
>>>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-10:)
>>>>>>> Execution is successful.
>>>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>>>> Sending ping: Seq 788-8:  { Cmd , MgmtId: -1, via: 788, Ver:
v1, Flags:
>>>>>>> 11,
>>>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>>>> 18.domaina.net
>>>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (Agent-Handler-3:)
>>>>>>> Received response: Seq 788-8:  { Ans: , MgmtId: 161344838950,
via: 788,
>>>>>>> Ver: v1, Flags: 100010,
>>>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-3:) Processing command:
>>>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top
-b -n
>>>>>>> 1|grep
>>>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>>>> '{print $2}'
>>>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-3:) Execution is successful.
>>>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-4:) Processing command:
>>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-5:) Processing command:
>>>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Execution is successful.
>>>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-11:)
>>>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>>>>>> libvirt,
>>>>>>> continuing
>>>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-11:)
>>>>>>> Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>>>>>> -i
>>>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>>>>>> (Thread-11:)
>>>>>>> Execution is successful.
>>>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:)
>>>>>>> Sending ping: Seq 788-9:  { Cmd , MgmtId: -1, via: 788, Ver:
v1, Flags:
>>>>>>> 11,
>>>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>>>>>> 18.domaina.net
>>>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (Agent-Handler-2:)
>>>>>>> Received response: Seq 788-9:  { Ans: , MgmtId: 161344838950,
via: 788,
>>>>>>> Ver: v1, Flags: 100010,
>>>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>>>>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>>>>>> }
>>>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Executing:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>
>>>>>>> 2015-03-10 16:45:36,684{GMT} WARN  [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (Script-5:) Interrupting script.
>>>>>>> 2015-03-10 16:45:36,686{GMT} WARN  [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) Timed out:
>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>> hviasocket.pl
>>>>>>> -n
>>>>>>> r-2191-VM -p %vpccidr=
>>>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>> .  Output is:
>>>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-2:) Processing command:
>>>>>>> com.cloud.agent.api.GetHostStatsCommand
>>>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top
-b -n
>>>>>>> 1|grep
>>>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep
>>>>>>> cache:|awk '{print $4}');echo $freeMem
>>>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk
>>>>>>> '{print $2}'
>>>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>>>>>> LibvirtComputingResource]
>>>>>>> (agentRequest-Handler-2:) Execution is successful.
>>>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Trying to connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Could not connect to 169.254.2.99
>>>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>>>>>> (agentRequest-Handler-3:) Processing command:
>>>>>>> com.cloud.agent.api.GetVmStatsCommand
>>>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>>> (agentRequest-Handler-1:)
>>>>>>> Unable to logon to 169.254.2.99
>>>>>>>
>>>>>>> On 10 March 2015 at 17:50, ilya musayev <ilya.mailing.lists@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>   What kernel are you running? Please include Libvirt and Qemu
version.
>>>>>>>
>>>>>>>>
>>>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>>>>>>>
>>>>>>>>   Hi,
>>>>>>>>
>>>>>>>>> we are having issues from time to time with starting
VR - while
>>>>>>>>> creating
>>>>>>>>> VPC.
>>>>>>>>>
>>>>>>>>> Here is the few lines from MGMT log:
>>>>>>>>>
>>>>>>>>> 2015-03-10 16:11:50,018 WARN  [c.c.n.v.VpcManagerImpl]
>>>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to
start vpc [VPC
>>>>>>>>> [621-VPC-delete] due to
>>>>>>>>> com.cloud.exception.AgentUnavailableException: Resource
[Host:788]
>>>>>>>>> is
>>>>>>>>> unreachable: Host 788: Unable to start instance due to
Unable to
>>>>>>>>> start
>>>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart,
not
>>>>>>>>> retrying
>>>>>>>>>            at
>>>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>>>>>>>> VirtualMachineManagerImpl.java:1072)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The interesting part is on the Agent itself:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN  [kvm.resource.
>>>>>>>>> LibvirtComputingResource]
>>>>>>>>> (agentRequest-Handler-2:) Timed out:
>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>>>>>>>> hviasocket.pl
>>>>>>>>> -n
>>>>>>>>> r-2189-VM -p %vpccidr=
>>>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>>>>>>>> .  Output is:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4
 out of 5
>>>>>>>>> works.
>>>>>>>>>
>>>>>>>>> Any suggestion are really appreciated, since this is
live system :(
>>>>>>>>> ACS 4.3.0, KVM, CEPH
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić

Mime
View raw message