Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5E06F18907 for ; Tue, 8 Mar 2016 14:05:44 +0000 (UTC) Received: (qmail 61585 invoked by uid 500); 8 Mar 2016 14:05:43 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 61444 invoked by uid 500); 8 Mar 2016 14:05:41 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 61316 invoked by uid 500); 8 Mar 2016 14:05:41 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 61312 invoked by uid 99); 8 Mar 2016 14:05:41 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2016 14:05:41 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id E91DB2C044E for ; Tue, 8 Mar 2016 14:05:40 +0000 (UTC) Date: Tue, 8 Mar 2016 14:05:40 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect to XEN and KVM hosts MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184938#comment-15184938 ] ASF GitHub Bot commented on CLOUDSTACK-9285: -------------------------------------------- Github user kiwiflyer commented on the pull request: https://github.com/apache/cloudstack/pull/1430#issuecomment-193796033 @ustcweizhou So after looking at our issue and the one reported originally in 9285, I think they're two different issues. One related to initial connection on agent startup and our one, related to a re-connection event where the connection is terminated unexpectedly. > Cloudstack 4.8 can't connect to XEN and KVM hosts > ------------------------------------------------- > > Key: CLOUDSTACK-9285 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: KVM, Xen > Affects Versions: 4.8.0 > Environment: CentOS 7 > Reporter: Jens Fettig > Priority: Blocker > Attachments: agentfailure.pcap > > > We used Cloudstack 4.7 in our testing environment. For future uses we updated to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 cloudstack has some problems to connect to the hosts again. Here are the error logs. > {code:title=XEN Server from the managment-server.log:|borderStyle=solid} > 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly connected host 13(xen5) > 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is running... > 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is running... > 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is running... > 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is running... > 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary storage VM for secondary storage host cdsdev-secondary > 2016-02-15 11:22:11,018 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired async-jobs > 2016-02-15 11:22:11,024 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired async-jobs > 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 13, name = xen5] > 2016-02-15 11:22:11,101 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) can not load directly connected host 13(xen5) due to > com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > at com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163) > at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523) > at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827) > at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158) > at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697) > at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217) > at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182) > at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96) > at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233) > at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > at com.xensource.xenapi.Types.checkResponse(Types.java:322) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at com.xensource.xenapi.Session.loginWithPassword(Session.java:587) > ... 19 more > {code} > This seems to be the problem: > Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > I updated the vhd-utils like in the upgrade gourde from 4.6: > http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.7.0/upgrade/upgrade-4.6.html > But maybe a need a new vhd-utils file for 4.8, because this one was for 4.7? > {code:title=KVM from the agent.log:|borderStyle=solid} > 2016-02-15 10:46:20,971 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started > 2016-02-15 10:46:20,973 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.8.0 > 2016-02-15 10:46:20,974 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > 2016-02-15 10:46:20,977 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage > 2016-02-15 10:46:20,978 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm > 2016-02-15 10:46:20,988 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > 2016-02-15 10:46:20,999 INFO [cloud.agent.AgentShell] (main:null) (logid:) Preferring IPv4 address family for agent connection > 2016-02-15 10:46:21,095 INFO [cloud.agent.Agent] (main:null) (logid:) id is 17 > 2016-02-15 10:46:21,116 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one > 2016-02-15 10:46:21,373 INFO [org.reflections.Reflections] (main:null) (logid:) Reflections took 66 ms to scan 1 urls, producing 7 keys and 10 values > 2016-02-15 10:46:21,514 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver. > 2016-02-15 10:46:21,543 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 17 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.0.72 : port = 8250 > 2016-02-15 10:46:21,549 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.0.72:8250 > 2016-02-15 10:46:21,609 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads. > java.io.IOException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.Link.doHandshake(Link.java:513) > at com.cloud.utils.nio.NioClient.init(NioClient.java:80) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88) > at com.cloud.agent.Agent.start(Agent.java:228) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399) > at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) > at com.cloud.agent.AgentShell.start(AgentShell.java:461) > at com.cloud.agent.AgentShell.main(AgentShell.java:496) > 2016-02-15 10:46:21,611 INFO [utils.exception.CSExceptionErrorCode] (main:null) (logid:) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions > 2016-02-15 10:46:21,612 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: > com.cloud.utils.exception.CloudRuntimeException: Unable to start the connection! > at com.cloud.agent.Agent.start(Agent.java:230) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399) > at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) > at com.cloud.agent.AgentShell.start(AgentShell.java:461) > at com.cloud.agent.AgentShell.main(AgentShell.java:496) > Caused by: com.cloud.utils.exception.NioConnectionException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:94) > at com.cloud.agent.Agent.start(Agent.java:228) > ... 5 more > Caused by: java.io.IOException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.Link.doHandshake(Link.java:513) > at com.cloud.utils.nio.NioClient.init(NioClient.java:80) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88) > ... 6 more > {code} > cloudstack-agent on the kvm host and the cloudstack-management on the cloudstack server is running. Everything on version 4.8.0. > Maybe there are some bugs or did I something wrong? > Greetings J. Fettig -- This message was sent by Atlassian JIRA (v6.3.4#6332)