cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rayees Namathponnan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CLOUDSTACK-5432) [Automation] Libvtd getting crashed and agent going to alert start
Date Tue, 10 Dec 2013 04:56:06 GMT
Rayees Namathponnan created CLOUDSTACK-5432:
-----------------------------------------------

             Summary: [Automation] Libvtd getting crashed and agent going to alert start 
                 Key: CLOUDSTACK-5432
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5432
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: KVM
    Affects Versions: 4.3.0
         Environment: KVM (RHEL 6.3)
Branch : 4.3
            Reporter: Rayees Namathponnan
             Fix For: 4.3.0


This issue is observed in  4.3 automation environment;  libvirt crashed and cloudstack agent
went to alert start;

Please see the agent log; connection between agent and MS lost with error "Connection closed
with -1 on reading size."  @ 2013-12-09 19:47:06,969

2013-12-09 19:43:41,495 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing
command: com.cloud.agent.api.GetStorageStatsCommand
2013-12-09 19:47:06,969 DEBUG [utils.nio.NioConnection] (Agent-Selector:null) Location 1:
Socket Socket[addr=/10.223.49.195,port=8250,localport=40801] closed on read.  Probably -1
returned: Connection closed with -1 on reading size.
2013-12-09 19:47:06,969 DEBUG [utils.nio.NioConnection] (Agent-Selector:null) Closing socket
Socket[addr=/10.223.49.195,port=8250,localport=40801]
2013-12-09 19:47:06,969 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) Clearing watch list:
2
2013-12-09 19:47:11,969 INFO  [cloud.agent.Agent] (Agent-Handler-3:null) Lost connection to
the server. Dealing with the remaining commands...
2013-12-09 19:47:11,970 INFO  [cloud.agent.Agent] (Agent-Handler-3:null) Cannot connect because
we still have 5 commands in progress.
2013-12-09 19:47:16,970 INFO  [cloud.agent.Agent] (Agent-Handler-3:null) Lost connection to
the server. Dealing with the remaining commands...
2013-12-09 19:47:16,990 INFO  [cloud.agent.Agent] (Agent-Handler-3:null) Cannot connect because
we still have 5 commands in progress.
2013-12-09 19:47:21,990 INFO  [cloud.agent.Agent] (Agent-Handler-3:null) Lost connection to
the server. Dealing with the remaining commands.. 

Please see the lib virtd log at same time (please see the attached complete log, there is
a 5 hour  difference in agent log and libvirt log ) 


2013-12-10 02:45:45.563+0000: 5938: error : qemuMonitorIO:574 : internal error End of file
from monitor
2013-12-10 02:45:47.663+0000: 5942: error : virCommandWait:2308 : internal error Child process
(/bin/umount /mnt/41b632b5-40b3-3024-a38b-ea259c72579f) status unexpected: exit status 16
2013-12-10 02:45:53.925+0000: 5943: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet14 root) status unexpected: exit status 2
2013-12-10 02:45:53.929+0000: 5943: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet14 ingress) status unexpected: exit status 2
2013-12-10 02:45:54.011+0000: 5943: warning : qemuDomainObjTaint:1297 : Domain id=71 name='i-45-97-QA'
uuid=7717ba08-be84-4b63-a674-1534f9dc7bef is tainted: high-privileges
2013-12-10 02:46:33.070+0000: 5940: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet12 root) status unexpected: exit status 2
2013-12-10 02:46:33.081+0000: 5940: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet12 ingress) status unexpected: exit status 2
2013-12-10 02:46:33.197+0000: 5940: warning : qemuDomainObjTaint:1297 : Domain id=72 name='i-47-111-QA'
uuid=7fcce58a-96dc-4207-9998-b8fb72b446ac is tainted: high-privileges
2013-12-10 02:46:36.394+0000: 5938: error : qemuMonitorIO:574 : internal error End of file
from monitor
2013-12-10 02:46:37.685+0000: 5940: error : virCommandWait:2308 : internal error Child process
(/bin/umount /mnt/41b632b5-40b3-3024-a38b-ea259c72579f) status unexpected: exit status 16
2013-12-10 02:46:57.869+0000: 5940: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet15 root) status unexpected: exit status 2
2013-12-10 02:46:57.873+0000: 5940: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet15 ingress) status unexpected: exit status 2
2013-12-10 02:46:57.925+0000: 5940: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet17 root) status unexpected: exit status 2
2013-12-10 02:46:57.933+0000: 5940: error : virCommandWait:2308 : internal error Child process
(/sbin/tc qdisc del dev vnet17 ingress) status unexpected: exit status 2
2013-12-10 02:46:58.034+0000: 5940: warning : qemuDomainObjTaint:1297 : Domain id=73 name='r-114-QA'
uuid=8ded6f1b-69e7-419d-8396-5795372d0ae2 is tainted: high-privileges
2013-12-10 02:47:22.762+0000: 5938: error : qemuMonitorIO:574 : internal error End of file
from monitor
2013-12-10 02:47:23.273+0000: 5939: error : virCommandWait:2308 : internal error Child process
(/bin/umount /mnt/41b632b5-40b3-3024-a38b-ea259c72579f) status unexpected: exit status 16


virsh command doest not return anything and hung;
[root@Rack2Host11 libvirt]# virsh list


Work around

If i restart libvirtd,  agent can connect MS 




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message