Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C9A8D1939D for ; Thu, 24 Mar 2016 15:13:26 +0000 (UTC) Received: (qmail 98052 invoked by uid 500); 24 Mar 2016 15:13:26 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 98027 invoked by uid 500); 24 Mar 2016 15:13:26 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 98006 invoked by uid 500); 24 Mar 2016 15:13:26 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 97997 invoked by uid 99); 24 Mar 2016 15:13:26 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2016 15:13:26 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 6563B2C14DC for ; Thu, 24 Mar 2016 15:13:26 +0000 (UTC) Date: Thu, 24 Mar 2016 15:13:26 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect to XEN and KVM hosts MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210369#comment-15210369 ] ASF subversion and git services commented on CLOUDSTACK-9285: ------------------------------------------------------------- Commit 4db1c0125a3b8f5c46a9e3500fe3eec4ca9e95dc in cloudstack's branch refs/heads/4.7 from [~wstevens] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=4db1c01 ] Merge pull request #1430 from myENA/4.7_cloudstack-9285 CLOUDSTACK-9285 for 4.7.xPer Daan's request, here is a pull request for the 4.7.x release. * pr/1430: Cloudstack 9285 for 4.7.x CLOUDSTACK-9285 - Address original on start exception(s) and newline cleanup Cloudstack 9285 for 4.7.x Signed-off-by: Will Stevens > Cloudstack 4.8 can't connect to XEN and KVM hosts > ------------------------------------------------- > > Key: CLOUDSTACK-9285 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: KVM, Xen > Affects Versions: 4.8.0 > Environment: CentOS 7 > Reporter: Jens Fettig > Priority: Blocker > Attachments: agentfailure.pcap > > > We used Cloudstack 4.7 in our testing environment. For future uses we updated to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 cloudstack has some problems to connect to the hosts again. Here are the error logs. > {code:title=XEN Server from the managment-server.log:|borderStyle=solid} > 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly connected host 13(xen5) > 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is running... > 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is running... > 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is running... > 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is running... > 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary storage VM for secondary storage host cdsdev-secondary > 2016-02-15 11:22:11,018 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired async-jobs > 2016-02-15 11:22:11,024 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired async-jobs > 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 13, name = xen5] > 2016-02-15 11:22:11,101 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) can not load directly connected host 13(xen5) due to > com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > at com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163) > at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523) > at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827) > at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158) > at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697) > at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217) > at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182) > at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96) > at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233) > at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > at com.xensource.xenapi.Types.checkResponse(Types.java:322) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at com.xensource.xenapi.Session.loginWithPassword(Session.java:587) > ... 19 more > {code} > This seems to be the problem: > Unable to create master connection to host(192.168.0.97) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle. > I updated the vhd-utils like in the upgrade gourde from 4.6: > http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.7.0/upgrade/upgrade-4.6.html > But maybe a need a new vhd-utils file for 4.8, because this one was for 4.7? > {code:title=KVM from the agent.log:|borderStyle=solid} > 2016-02-15 10:46:20,971 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started > 2016-02-15 10:46:20,973 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.8.0 > 2016-02-15 10:46:20,974 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > 2016-02-15 10:46:20,977 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage > 2016-02-15 10:46:20,978 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm > 2016-02-15 10:46:20,988 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > 2016-02-15 10:46:20,999 INFO [cloud.agent.AgentShell] (main:null) (logid:) Preferring IPv4 address family for agent connection > 2016-02-15 10:46:21,095 INFO [cloud.agent.Agent] (main:null) (logid:) id is 17 > 2016-02-15 10:46:21,116 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one > 2016-02-15 10:46:21,373 INFO [org.reflections.Reflections] (main:null) (logid:) Reflections took 66 ms to scan 1 urls, producing 7 keys and 10 values > 2016-02-15 10:46:21,514 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver. > 2016-02-15 10:46:21,543 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 17 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 192.168.0.72 : port = 8250 > 2016-02-15 10:46:21,549 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.0.72:8250 > 2016-02-15 10:46:21,609 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads. > java.io.IOException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.Link.doHandshake(Link.java:513) > at com.cloud.utils.nio.NioClient.init(NioClient.java:80) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88) > at com.cloud.agent.Agent.start(Agent.java:228) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399) > at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) > at com.cloud.agent.AgentShell.start(AgentShell.java:461) > at com.cloud.agent.AgentShell.main(AgentShell.java:496) > 2016-02-15 10:46:21,611 INFO [utils.exception.CSExceptionErrorCode] (main:null) (logid:) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions > 2016-02-15 10:46:21,612 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: > com.cloud.utils.exception.CloudRuntimeException: Unable to start the connection! > at com.cloud.agent.Agent.start(Agent.java:230) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399) > at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) > at com.cloud.agent.AgentShell.start(AgentShell.java:461) > at com.cloud.agent.AgentShell.main(AgentShell.java:496) > Caused by: com.cloud.utils.exception.NioConnectionException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:94) > at com.cloud.agent.Agent.start(Agent.java:228) > ... 5 more > Caused by: java.io.IOException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.Link.doHandshake(Link.java:513) > at com.cloud.utils.nio.NioClient.init(NioClient.java:80) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88) > ... 6 more > {code} > cloudstack-agent on the kvm host and the cloudstack-management on the cloudstack server is running. Everything on version 4.8.0. > Maybe there are some bugs or did I something wrong? > Greetings J. Fettig -- This message was sent by Atlassian JIRA (v6.3.4#6332)