Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7046F11C05 for ; Tue, 22 Apr 2014 22:58:16 +0000 (UTC) Received: (qmail 11557 invoked by uid 500); 22 Apr 2014 22:58:15 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 11532 invoked by uid 500); 22 Apr 2014 22:58:15 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 11522 invoked by uid 500); 22 Apr 2014 22:58:14 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 11518 invoked by uid 99); 22 Apr 2014 22:58:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Apr 2014 22:58:14 +0000 Date: Tue, 22 Apr 2014 22:58:14 +0000 (UTC) From: "Rayees Namathponnan (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CLOUDSTACK-6475) [Automation] communication between cloudstack agent and MS disconnecting continuously MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rayees Namathponnan updated CLOUDSTACK-6475: -------------------------------------------- Attachment: management-server.rar > [Automation] communication between cloudstack agent and MS disconnecting continuously > --------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-6475 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6475 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: KVM > Affects Versions: 4.4.0 > Environment: RHEL 6.3 > Reporter: Rayees Namathponnan > Priority: Blocker > Fix For: 4.4.0 > > Attachments: management-server.rar > > > This issue is observed during automation, run. > communication between cloudstack agent and ms getting disconnected continuously; observed below error in agent log > 2014-04-22 04:08:47,867 INFO [cloud.agent.Agent] (AgentShutdownThread:null) Stopping the agent: Reason = sig.kill > 2014-04-22 04:10:41,456 INFO [cloud.agent.AgentShell] (main:null) Agent started > 2014-04-22 04:10:41,500 INFO [cloud.agent.AgentShell] (main:null) Implementation Version is 4.4.0-SNAPSHOT > 2014-04-22 04:10:41,502 INFO [cloud.agent.AgentShell] (main:null) agent.properties found at /etc/cloudstack/agent/agent.properties > 2014-04-22 04:10:41,551 INFO [cloud.agent.AgentShell] (main:null) Defaulting to using properties file for storage > 2014-04-22 04:10:41,552 INFO [cloud.agent.AgentShell] (main:null) Defaulting to the constant time backoff algorithm > 2014-04-22 04:10:41,572 INFO [cloud.utils.LogUtils] (main:null) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > 2014-04-22 04:10:41,722 INFO [cloud.agent.Agent] (main:null) id is 0 > 2014-04-22 04:10:42,501 INFO [kvm.resource.LibvirtComputingResource] (main:null) No libvirt.vif.driver specified. Defaults to BridgeVifDriver. > 2014-04-22 04:10:42,590 INFO [cloud.agent.Agent] (main:null) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 10.223.49.195 : port = 8250 > 2014-04-22 04:10:42,664 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to 10.223.49.195:8250 > 2014-04-22 04:10:42,920 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done > 2014-04-22 04:10:42,920 INFO [utils.nio.NioClient] (Agent-Selector:null) Connected to 10.223.49.195:8250 > 2014-04-22 04:10:42,941 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Could not read cpuinfo_max_freq > 2014-04-22 04:10:43,158 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) Default Builder inited. > 2014-04-22 04:10:43,227 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Proccess agent startup answer, agent id = 0 > 2014-04-22 04:10:43,227 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Set agent id 0 > 2014-04-22 04:10:43,233 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Startup Response Received: agent id = 0 > 2014-04-22 04:11:40,925 INFO [cloud.agent.Agent] (Agent-Handler-1:null) Lost connection to the server. Dealing with the remaining commands... > 2014-04-22 04:11:42,352 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 0 > 2014-04-22 04:11:42,368 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 1 > 2014-04-22 04:11:42,383 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 2 > 2014-04-22 04:11:42,398 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 3 > 2014-04-22 04:11:42,413 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 4 > 2014-04-22 04:11:42,414 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error; reboot the host > 2014-04-22 04:11:42,472 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 0 > 2014-04-22 04:11:42,487 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 1 > 2014-04-22 04:11:42,507 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 2 > 2014-04-22 04:11:42,527 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 3 > 2014-04-22 04:11:42,542 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error, retry: 4 > 2014-04-22 04:11:42,542 WARN [kvm.resource.KVMHAMonitor] (Thread-4:null) write heartbeat failed: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line 131: echo: write error: Input/output error; reboot the host > 2014-04-22 04:11:45,926 INFO [cloud.agent.Agent] (Agent-Handler-1:null) Reconnecting... > 2014-04-22 04:11:45,927 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to 10.223.49.195:8250 > 2014-04-22 04:11:45,929 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads. > java.net.SocketException: Network is unreachable > at sun.nio.ch.Net.connect0(Native Method) > at sun.nio.ch.Net.connect(Net.java:465) > at sun.nio.ch.Net.connect(Net.java:457) > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666) > at com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at com.cloud.utils.nio.NioConnection.run(NioConnection.java:111) > at java.lang.Thread.run(Thread.java:744) > 2014-04-22 04:11:46,789 INFO [cloud.agent.Agent] (AgentShutdownThread:null) Stopping the agent: Reason = sig.kill > 2014-04-22 04:13:43,347 INFO [cloud.agent.AgentShell] (main:null) Agent started > 2014-04-22 04:13:43,392 INFO [cloud.agent.AgentShell] (main:null) Implementation Version is 4.4.0-SNAPSHOT > 2014-04-22 04:13:43,394 INFO [cloud.agent.AgentShell] (main:null) agent.properties found at /etc/cloudstack/agent/agent.properties > 2014-04-22 04:13:43,442 INFO [cloud.agent.AgentShell] (main:null) Defaulting to using properties file for storage > 2014-04-22 04:13:43,444 INFO [cloud.agent.AgentShell] (main:null) Defaulting to the constant time backoff algorithm > 2014-04-22 04:13:43,464 INFO [cloud.utils.LogUtils] (main:null) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > 2014-04-22 04:13:43,616 INFO [cloud.agent.Agent] (main:null) id is 0 > 2014-04-22 04:13:44,442 INFO [kvm.resource.LibvirtComputingResource] (main:null) No libvirt.vif.driver specified. Defaults to BridgeVifDriver. > 2014-04-22 04:13:44,539 INFO [cloud.agent.Agent] (main:null) Agent [id = 0 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 : host = 10.223.49.195 : port = 8250 > 2014-04-22 04:13:44,606 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to 10.223.49.195:8250 > 2014-04-22 04:13:44,869 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done > 2014-04-22 04:13:44,870 INFO [utils.nio.NioClient] (Agent-Selector:null) Connected to 10.223.49.195:8250 > 2014-04-22 04:13:44,892 WARN [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null) Could not read cpuinfo_max_freq > 2014-04-22 04:13:45,099 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) Default Builder inited. > 2014-04-22 04:13:45,166 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Proccess agent startup answer, agent id = 0 > 2014-04-22 04:13:45,166 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Set agent id 0 > 2014-04-22 04:13:45,174 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Startup Response Received: agent id = 0 > 2014-04-22 04:14:43,163 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Lost connection to the server. Dealing with the remaining commands... > 2014-04-22 04:14:48,164 INFO [cloud.agent.Agent] (Agent-Handler-2:null) Reconnecting... > 2014-04-22 04:14:48,164 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to 10.223.49.195:8250 > 2014-04-22 04:14:48,257 INFO [utils.nio.NioClient] (Agent-Selector:null) SSL: Handshake done > 2014-04-22 04:14:48,257 INFO [utils.nio.NioClient] (Agent-Selector:null) Connected to 10.223.49.195:8250 > -- This message was sent by Atlassian JIRA (v6.2#6252)