cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-9590) KVM + CentOS 7.2 + Agent in Alert State for long time
Date Fri, 11 Nov 2016 09:06:58 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656573#comment-15656573
] 

Wei Zhou commented on CLOUDSTACK-9590:
--------------------------------------

key logs
{code}
2016-11-10 13:23:06,568 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null) (logid:) Seq
-1-4: Scheduling the first command  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":2,"cpus":8,"speed":2261,"memory":100092813312,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-327.36.3.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.2.1511","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"3","pod":"3","cluster":"9","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","iqn":"iqn.1994-05.com.redhat:eedee56bd952","privateIpAddress":"192.168.85.14","privateMacAddress":"52:80:f7:fc:af:42","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.85.14","storageNetmask":"255.255.255.0","storageMacAddress":"52:80:f7:fc:af:42","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.85.254","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"b3b9ef96-18b1-4136-8a69-5b316c6dc123","host":"192.168.85.14","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":30866534400,"availableBytes":29248552960},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","resourceName":"LibvirtComputingResource","wait":0}}]
}
2016-11-10 13:23:06,570 INFO  [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:)
Connection from /192.168.85.14 closed but no cleanup was done.
2016-11-10 13:23:06,582 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-15:null) (logid:)
Failed to send startupanswer: java.nio.channels.ClosedChannelException
2016-11-10 13:23:06,605 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb)
Seq -1-4: Processing the first command  { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 1, [{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":2,"cpus":8,"speed":2261,"memory":100092813312,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-327.36.3.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.2.1511","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"3","pod":"3","cluster":"9","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","iqn":"iqn.1994-05.com.redhat:eedee56bd952","privateIpAddress":"192.168.85.14","privateMacAddress":"52:80:f7:fc:af:42","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.85.14","storageNetmask":"255.255.255.0","storageMacAddress":"52:80:f7:fc:af:42","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.85.254","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"b3b9ef96-18b1-4136-8a69-5b316c6dc123","host":"192.168.85.14","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":30866534400,"availableBytes":29248552960},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","resourceName":"LibvirtComputingResource","wait":0}}]
}
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BareMetalDiscoverer
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to BrocadeVcsElement
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer
2016-11-10 13:23:06,627 DEBUG [c.c.h.o.r.Ovm3Discoverer] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) createHostVOForConnectedAgent: Host[-51-Routing]
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to LxcServerDiscoverer
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to Ovs
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to OvmDiscoverer
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer
2016-11-10 13:23:06,687 DEBUG [c.c.r.ResourceState] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Resource state update: [id = 51; name = kvm02.oscloud.local; old state =
Enabled; event = InternalCreated; new state = Enabled]
2016-11-10 13:23:06,688 DEBUG [c.c.h.Status] (AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb)
Transition:[Resource state = Enabled, Agent event = AgentConnected, Host id = 51, name = kvm02.oscloud.local]
2016-11-10 13:23:06,705 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) create ClusteredAgentAttache for 51
2016-11-10 13:23:06,709 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Sending Connect to listener: XcpServerDiscoverer
2016-11-10 13:23:06,709 DEBUG [c.c.h.x.d.XcpServerDiscoverer] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Not XenServer so moving on.
2016-11-10 13:23:06,709 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Sending Connect to listener: HypervServerDiscoverer
2016-11-10 13:23:06,709 DEBUG [c.c.h.h.d.HypervServerDiscoverer] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Not Hyper-V hypervisor, so moving on.
2016-11-10 13:23:06,709 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Sending Connect to listener: SecurityGroupListener
2016-11-10 13:23:06,709 INFO  [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Received a host startup notification
2016-11-10 13:23:06,714 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb)
Seq 51-746753113213370369: Sending  { Cmd , MgmtId: 3232257305, via: 51(kvm02.oscloud.local),
Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":2382,"wait":0}}]
}
2016-11-10 13:23:06,714 INFO  [c.c.a.m.AgentAttache] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Seq 51-746753113213370369: Unable to send due to Resource [Host:51] is unreachable:
Host 51: Channel is closed
2016-11-10 13:23:06,714 DEBUG [c.c.a.m.AgentAttache] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Seq 51-746753113213370369: Cancelling.
2016-11-10 13:23:06,714 DEBUG [c.c.n.s.SecurityGroupListener] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Unable to schedule network rules cleanup for host 51
com.cloud.exception.AgentUnavailableException: Resource [Host:51] is unreachable: Host 51:
Channel is closed
	at com.cloud.agent.manager.ConnectedAgentAttache.send(ConnectedAgentAttache.java:46)
	at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:373)
	at com.cloud.agent.manager.ClusteredAgentAttache.send(ClusteredAgentAttache.java:141)
	at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:507)
	at com.cloud.network.security.SecurityGroupListener.processConnect(SecurityGroupListener.java:169)
	at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:564)
	at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1087)
	at com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:120)
	at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1171)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
{code}

> KVM + CentOS 7.2 + Agent in Alert State for long time
> -----------------------------------------------------
>
>                 Key: CLOUDSTACK-9590
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9590
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: cloudstack-agent
>    Affects Versions: 4.9.0
>         Environment: entOS Linux release 7.2.1511 (Core)
> cloudstack-agent-4.9.0-1.el7.centos.x86_64
>            Reporter: Sven Vogel
>         Attachments: agent.log, cloudstack-startup.log, management-server.zip
>
>
> Hi,
> When i add a new host to cloudstack management server it take some time to get host out
from alert state.
> 1. i add the host and host add not possible
> 2. values are correct set to agent.properties, restart cloustack agent
> 3. agent says connected to server
> 4. management server says "alert"
> management-server.log
> 2016-11-10 13:23:06,783 DEBUG [c.c.h.Status] (AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb)
Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host
> id = 51, name = kvm02.oscloud.local]
> 2016-11-10 13:23:06,798 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Notifying other nodes of to disconnect
> 2016-11-10 13:23:06,806 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-49:ctx-c3b72839)
(logid:5a86e1fb) Failed to handle host connection: com.cloud.exception.Connection
> Exception: Unable to get an answer to the CheckNetworkCommand from agent: 51
> is there any way to speed up the alert state? is it normal that it take so long?
> thanks
> Sven



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message