cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milamber (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-9255) Unable to start VM DomainRouter due to error in finalizeStart, not retrying
Date Sat, 23 Jan 2016 23:33:39 GMT
Milamber created CLOUDSTACK-9255:
------------------------------------

             Summary: Unable to start VM DomainRouter due to error in finalizeStart, not retrying
                 Key: CLOUDSTACK-9255
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9255
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Virtual Router
    Affects Versions: 4.7.0, 4.6.2, 4.8.0, 4.7.1
         Environment: Ubuntu 14.04.3
KVM
NFS (primary/secondary)
            Reporter: Milamber



I've spent 3 days with the same issue : unable to restart with clean up a network (virtual
router or redondant virtual router) if the network have at least 20 virtual machines.

I've tested with CS 4.6.2, 4.7.0, 4.7.1RC1, 4.8.0RC1, same problem. I've used the system vm
from apt-get.eu and last builds from jenkins.

My tests are made with hosts/mgr on Ubuntu 14.04.3 / KVM / NFS primary/secondary.

My test case (with ansible modules) :
1/ create a new network (normal or RVR)
2/ create 20 vms (same params, just the name is changes)
wait the end of creation
3/ restart the network with clean up option
4/ wait the restart, after some minutes, an error message arrived : "Failed to restart network"

The trace in management.log are:

2016-01-23 23:02:51,503 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-51:ctx-9ed51622
job-268/job-271) (logid:b9a521fa) Unable to complete AsyncJobVO {id:271, userId: 2, accountId:
2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAMnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AA5SZXN0YXJ0TmV0d29ya3QAP3JPMEFCWE55QUJGcVlYWmhMbXhoYm1jdVFtOXZiR1ZoYnMwZ2NvRFZuUHJ1QWdBQldnQUZkbUZzZFdWNGNBRXhw,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
146456419427, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sat Jan 23
22:56:00 CET 2016}, job origin:268
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable
to start instance due to Unable to start VM[DomainRouter|r-50-VM] due to error in finalizeStart,
not retrying
    at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
    at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4578)
    at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
    at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4734)
    at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
    at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
    at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM[DomainRouter|r-50-VM]
due to error in finalizeStart, not retrying
    at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1083)
    at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4578)
    at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
    ... 17 more


During the restart of the network I can connect on the VR with link local link over ssh, the
last lines shows:

2016-01-23 22:02:39,780  configure.py __init__:128 AclIP created for rule ==> {'last_port':
65535, u'protocol': u'tcp', u'revoked': False, u'already_added': True, u'source_cidr_list':
[u'0.0.0.0/0'], 'cidr': [u'0.0.0.0/0'], u'id': 52, u'src_ip': u'192.168.13.30', u'purpose':
u'Firewall', 'allowed': True, 'action': 'ACCEPT', u'src_port_range': [1, 65535], u'traffic_type':
u'Ingress', 'type': u'tcp', u'default_egress_policy': False, 'first_port': 1}
2016-01-23 22:02:39,780  configure.py add_rule:165 Current ACL IP direction is ==> ingress
2016-01-23 22:02:39,780  merge.py load:60 Loading data bag type forwardingrules

Broadcast message from root@r-50-VM (Sat Jan 23 22:02:45 2016):

The system is going down for system halt NOW!

Broadcast message from root@r-50-VM (Sat Jan 23 22:02:45 2016):

Power button pressed
The system is going down for system halt NOW!
/opt/cloud/bin/vr_cfg.sh: line 60: 16845 Killed                  /opt/cloud/bin/update_config.py
vm_metadata.json
Sat Jan 23 22:02:46 UTC 2016 : VR config: executing failed: /opt/cloud/bin/update_config.py
vm_metadata.json
Connection to 169.254.2.186 closed by remote host.
Connection to 169.254.2.186 closed.



Perhaps that was a timeout issue? if I create one VM or 10 VMs, the network restart works.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message