cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Weller <>
Subject Re: Router VM: timeout issue on 1 out of 4 networks
Date Tue, 13 Dec 2016 02:31:41 GMT
Can you turn on agent debug mode and take a look at the debug level logs?

You can do that by running sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml on
the host and then restarting the agent.

- Si

From: Syahrul Sazli Shaharir <>
Sent: Monday, December 12, 2016 8:21 PM
Subject: Router VM: timeout issue on 1 out of 4 networks


I am running latest Cloudstack on CentOS 7 KVM + ceph
environment. After running for some time, I faced with an issue with
one out of 4 networks - following a heartbeat-induced reset on all
hosts, the associated virtual router would not get recreated and
started properly on any of the 3 hosts I have, even after repeated
attempts of the following:-
- destroy-recreate cycles, via Cloudstack UI
- restartNetwork cleanup=true API calls (failed with errorcode = 530).
- redownload and reregister system VM template as another entry and
assign to router VM in global setting (boots the new template OK, but
still same problem)
- tweak default system offering for router VM (increased RAM from 256 to 512MB)
- created new system offering, with RAM tweak, and use of ceph rbd
store, and assigned it to Cloud.Com-SoftwareRouter as per docs - which
didnt work for some reason: it kept on using initial default offering
and created image on local host storage
- upgrade to latest cloudstack (previously was running 4.8)

As with a handful of others in this list archives, virsh list and
dumpxml shows the VM created OK but failed soon after booting, as
found in the following error in agent.log :-

2016-12-13 10:03:33,894 WARN  [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) (logid:633e6e03) Timed out:
-n r-668-VM -p
.  Output is:
2016-12-13 10:05:45,895 WARN  [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) (logid:633e6e03) Timed out:
/usr/share/cloudstack-common/scripts/network/domr/ -c
/var/cache/cloud/VR-48ea8a95-6c02-499f-88d3-eae5bf9f9fbe.cfg .  Output

As mentioned, this only happens with 1 network (always the same
network). The other router VMs work OK. Any clues on how to
troubleshoot this further, would be greatly appreciated.


Syahrul Sazli Shaharir <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message