Return-Path: X-Original-To: apmail-cloudstack-issues-archive@www.apache.org Delivered-To: apmail-cloudstack-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6B6FC10FB6 for ; Tue, 7 Jan 2014 21:18:51 +0000 (UTC) Received: (qmail 40204 invoked by uid 500); 7 Jan 2014 21:18:51 -0000 Delivered-To: apmail-cloudstack-issues-archive@cloudstack.apache.org Received: (qmail 40175 invoked by uid 500); 7 Jan 2014 21:18:51 -0000 Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list issues@cloudstack.apache.org Received: (qmail 40112 invoked by uid 500); 7 Jan 2014 21:18:51 -0000 Delivered-To: apmail-incubator-cloudstack-issues@incubator.apache.org Received: (qmail 40067 invoked by uid 99); 7 Jan 2014 21:18:51 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jan 2014 21:18:51 +0000 Date: Tue, 7 Jan 2014 21:18:50 +0000 (UTC) From: "Animesh Chaturvedi (JIRA)" To: cloudstack-issues@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CLOUDSTACK-5432) [Automation] Libvtd getting crashed and agent going to alert start MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CLOUDSTACK-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864700#comment-13864700 ] Animesh Chaturvedi commented on CLOUDSTACK-5432: ------------------------------------------------ Rayees can you review and respond to Marcus > [Automation] Libvtd getting crashed and agent going to alert start > ------------------------------------------------------------------- > > Key: CLOUDSTACK-5432 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5432 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the default.) > Components: KVM > Affects Versions: 4.3.0 > Environment: KVM (RHEL 6.3) > Branch : 4.3 > Reporter: Rayees Namathponnan > Assignee: Marcus Sorensen > Priority: Blocker > Fix For: 4.3.0 > > Attachments: CLOUDSTACK-5432_Jan_06.rar, KVM_Automation_Dec_11.rar, agent1.rar, agent2.rar, management-server.rar > > > This issue is observed in 4.3 automation environment; libvirt crashed and cloudstack agent went to alert start; > Please see the agent log; connection between agent and MS lost with error "Connection closed with -1 on reading size." @ 2013-12-09 19:47:06,969 > 2013-12-09 19:43:41,495 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: com.cloud.agent.api.GetStorageStatsCommand > 2013-12-09 19:47:06,969 DEBUG [utils.nio.NioConnection] (Agent-Selector:null) Location 1: Socket Socket[addr=/10.223.49.195,port=8250,localport=40801] closed on read. Probably -1 returned: Connection closed with -1 on reading size. > 2013-12-09 19:47:06,969 DEBUG [utils.nio.NioConnection] (Agent-Selector:null) Closing socket Socket[addr=/10.223.49.195,port=8250,localport=40801] > 2013-12-09 19:47:06,969 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) Clearing watch list: 2 > 2013-12-09 19:47:11,969 INFO [cloud.agent.Agent] (Agent-Handler-3:null) Lost connection to the server. Dealing with the remaining commands... > 2013-12-09 19:47:11,970 INFO [cloud.agent.Agent] (Agent-Handler-3:null) Cannot connect because we still have 5 commands in progress. > 2013-12-09 19:47:16,970 INFO [cloud.agent.Agent] (Agent-Handler-3:null) Lost connection to the server. Dealing with the remaining commands... > 2013-12-09 19:47:16,990 INFO [cloud.agent.Agent] (Agent-Handler-3:null) Cannot connect because we still have 5 commands in progress. > 2013-12-09 19:47:21,990 INFO [cloud.agent.Agent] (Agent-Handler-3:null) Lost connection to the server. Dealing with the remaining commands.. > Please see the lib virtd log at same time (please see the attached complete log, there is a 5 hour difference in agent log and libvirt log ) > 2013-12-10 02:45:45.563+0000: 5938: error : qemuMonitorIO:574 : internal error End of file from monitor > 2013-12-10 02:45:47.663+0000: 5942: error : virCommandWait:2308 : internal error Child process (/bin/umount /mnt/41b632b5-40b3-3024-a38b-ea259c72579f) status unexpected: exit status 16 > 2013-12-10 02:45:53.925+0000: 5943: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet14 root) status unexpected: exit status 2 > 2013-12-10 02:45:53.929+0000: 5943: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet14 ingress) status unexpected: exit status 2 > 2013-12-10 02:45:54.011+0000: 5943: warning : qemuDomainObjTaint:1297 : Domain id=71 name='i-45-97-QA' uuid=7717ba08-be84-4b63-a674-1534f9dc7bef is tainted: high-privileges > 2013-12-10 02:46:33.070+0000: 5940: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet12 root) status unexpected: exit status 2 > 2013-12-10 02:46:33.081+0000: 5940: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet12 ingress) status unexpected: exit status 2 > 2013-12-10 02:46:33.197+0000: 5940: warning : qemuDomainObjTaint:1297 : Domain id=72 name='i-47-111-QA' uuid=7fcce58a-96dc-4207-9998-b8fb72b446ac is tainted: high-privileges > 2013-12-10 02:46:36.394+0000: 5938: error : qemuMonitorIO:574 : internal error End of file from monitor > 2013-12-10 02:46:37.685+0000: 5940: error : virCommandWait:2308 : internal error Child process (/bin/umount /mnt/41b632b5-40b3-3024-a38b-ea259c72579f) status unexpected: exit status 16 > 2013-12-10 02:46:57.869+0000: 5940: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet15 root) status unexpected: exit status 2 > 2013-12-10 02:46:57.873+0000: 5940: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet15 ingress) status unexpected: exit status 2 > 2013-12-10 02:46:57.925+0000: 5940: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet17 root) status unexpected: exit status 2 > 2013-12-10 02:46:57.933+0000: 5940: error : virCommandWait:2308 : internal error Child process (/sbin/tc qdisc del dev vnet17 ingress) status unexpected: exit status 2 > 2013-12-10 02:46:58.034+0000: 5940: warning : qemuDomainObjTaint:1297 : Domain id=73 name='r-114-QA' uuid=8ded6f1b-69e7-419d-8396-5795372d0ae2 is tainted: high-privileges > 2013-12-10 02:47:22.762+0000: 5938: error : qemuMonitorIO:574 : internal error End of file from monitor > 2013-12-10 02:47:23.273+0000: 5939: error : virCommandWait:2308 : internal error Child process (/bin/umount /mnt/41b632b5-40b3-3024-a38b-ea259c72579f) status unexpected: exit status 16 > virsh command doest not return anything and hung; > [root@Rack2Host11 libvirt]# virsh list > Work around > If i restart libvirtd, agent can connect MS -- This message was sent by Atlassian JIRA (v6.1.5#6160)