cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Witwicki <awitwi...@oakfordis.com>
Subject RE: Agent staying in alert mode
Date Fri, 20 Mar 2020 21:20:27 GMT
Restarting libvirtd helped

Thanks everyone

Adam

-----Original Message-----
From: Simon Weller <sweller@ena.com.INVALID>
Sent: 18 March 2020 14:31
To: users@cloudstack.apache.org
Subject: Re: Agent staying in alert mode

** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments
in this email with caution. **

No, it will not. All VMs will stay up.
________________________________
From: Adam Witwicki <awitwicki@oakfordis.com>
Sent: Wednesday, March 18, 2020 9:27 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: RE: Agent staying in alert mode

Will that stop all the Instances?

Thanks

Adam

-----Original Message-----
From: Simon Weller <sweller@ena.com.INVALID>
Sent: 18 March 2020 14:26
To: users@cloudstack.apache.org
Subject: Re: Agent staying in alert mode

** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments
in this email with caution. **

Have you tried restarting libvirtd and then restarting the cloudstack agent?

________________________________
From: Adam Witwicki <awitwicki@oakfordis.com>
Sent: Wednesday, March 18, 2020 3:06 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: RE: Agent staying in alert mode

Hi Andrija,

Setting the host to management IP, and not our LB VIP. The agent process seems to stall

2020-03-18 07:44:01,969 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null)
(logid:) cpus=32, speed=3800, ram=269385596928, _dom0MinMem=1073741824, _dom0OvercommitMem=0,
cpu sockets=2
2020-03-18 07:44:01,972 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:)
Parameters for private nic: 10.10.240.126 - a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:)
Parameters for storage nic: 10.10.240.126 - a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [cloud.resource.ServerResourceBase] (Agent-Handler-1:null) (logid:)
Parameters for public nic: 10.10.240.126 - a0:36:9f:24:32:6a-255.255.255.0
2020-03-18 07:44:01,973 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null)
(logid:) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/versions.sh
2020-03-18 07:44:01,984 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null)
(logid:) Execution is successful.
2020-03-18 07:44:01,985 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null)
(logid:) Executing: sudo grep InitiatorName= /etc/iscsi/initiatorname.iscsi
2020-03-18 07:44:01,992 DEBUG [kvm.resource.LibvirtComputingResource] (Agent-Handler-1:null)
(logid:) Execution is successful.
2020-03-18 07:44:01,993 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:)
Attempting to create storage pool 257c9140-3863-4be2-85e2-9835650f1b45 (Filesystem) in libvirt
2020-03-18 07:44:01,993 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:)
Looking for libvirtd connection at: qemu:///system HALTS HERE AND DOES NOT CONTINUE

I waited 10 mins for a timeout then stopped the agent

Thanks

Adam


-----Original Message-----
From: Andrija Panic <andrija.panic@gmail.com>
Sent: 17 March 2020 18:52
To: users <users@cloudstack.apache.org>
Subject: Re: Agent staying in alert mode

** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments
in this email with caution. **

Can you change that to the IP of the mgmt server and then restart the agent? it should always
point to the IP not hostname (what is that value there, "who put it there" etc - perhaps 
a valid
question?)

Best,


On Tue, 17 Mar 2020 at 15:38, Adam Witwicki <awitwicki@oakfordis.com> wrote:

> Hi Suresh
>
> /etc/cloudstack/agent/agent.properties
> host=cs.cloudstack
>
> telnet
> user@ois-mh-p1-c1-h2-b:~$ telnet cs.cloudstack 8250 Trying
> 10.10.240.10...
> Connected to cs.cloudstack.
> Escape character is '^]'.
>
>
> Same for all my other KVM nodes. 7 of 8 connected
>
> Thanks
>
> Adam
>
> -----Original Message-----
> From: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
> Sent: 17 March 2020 13:11
> To: users@cloudstack.apache.org
> Subject: Re: Agent staying in alert mode
>
> ** This mail originated from OUTSIDE the Oakford corporate network.
> Treat hyperlinks and attachments in this email with caution. **
>
> Hi Adam,
>
> What is the value of the host parameter in
> "/etc/cloudstack/agent/agent.properties" file of the KVM host with
> agent connectivity issue. Can you telnet to that management server on
> port 8250 from the KVM host.
>
> Are other KVM agents connected to same management server?
>
> Thanks,
> Suresh
>
> On Tue, Mar 17, 2020 at 3:05 PM Adam Witwicki
> <awitwicki@oakfordis.com>
> wrote:
>
> > Hi Suresh,
> >
> > Thanks for looking
> >
> > All KVM hosts can connect to management server - via ping
> >
> > The mgmt_server_id is Null for the alert host
> >
> > Host
> >
> > name;uuid;status;type;private_ip_address;private_netmask;private_mac
> > _a
> > ddress;storage_ip_address;storage_netmask;storage_mac_address;storag
> > e_
> > ip_address_2;storage_mac_address_2;storage_netmask_2;cluster_id;publ
> > ic
> > _ip_address;public_netmask;public_mac_address;proxy_port;data_center
> > _i
> > d;pod_id;cpu_sockets;cpus;speed;url;fs_type;hypervisor_type;hypervis
> > or
> > _version;ram;resource;version;parent;total_size;capabilities;guid;av
> > ai
> > lable;setup;dom0_memory;last_ping;mgmt_server_id;disconnected;create
> > d;
> > removed;update_count;resource_state;owner;lastUpdated;engine_state
> > ois-mh-p1-c1-h2-b;3d5e4c35-7e10-4560-8a2e-5ebf770e837d;Alert;Routing
> > ;1
> > 0.10.240.126;255.255.255.0;a0:36:9f:24:32:6a;10.10.240.126;255.255.2
> > 55
> > .0;a0:36:9f:24:32:6a;\N;\N;\N;5;10.10.240.126;255.255.255.0;a0:36:9f
> > :2
> > 4:32:6a;\N;5;5;2;32;3800;iqn.1993-08.org.debian:01:7c8e527ccba4;\N;K
> > VM
> > ;\N;269385596928;\N;4.11.0.0;\N;\N;hvm,snapshot;44190139-5b1b-3642-b
> > 78
> > 6-41ca577081e9-LibvirtComputingResource;1;0;0;1546450279;\N;2019-12-
> > 24
> > 09:25:53;2019-06-11 21:59:43;\N;2516;Enabled;\N;\N;Disabled
> >
> > mshost
> >
> > msid;runid;name;state;version;service_ip;service_port;last_update;re
> > mo
> > ved;alert_count
> > 91765225987;1584406841937;ois-mh-csmg-01;Up;4.11.0.0;10.10.240.21;90
> > 90
> > ;2020-03-17
> > 09:33:07;\N;0
> > 91765663747;1584406829845;ois-mh-csmg-02;Up;4.11.0.0;10.10.240.22;90
> > 90
> > ;2020-03-17
> > 09:33:09;\N;0
> > 91766390275;1584406828383;ois-mh-csmg-03;Up;4.11.0.0;10.10.240.23;90
> > 90
> > ;2020-03-17
> > 09:33:05;\N;0
> >
> >
> > Adam
> >
> > -----Original Message-----
> > From: Suresh Kumar Anaparti <sureshkumar.anaparti@gmail.com>
> > Sent: 16 March 2020 19:35
> > To: users@cloudstack.apache.org
> > Subject: Re: Agent staying in alert mode
> >
> > ** This mail originated from OUTSIDE the Oakford corporate network.
> > Treat hyperlinks and attachments in this email with caution. **
> >
> > Hi Adam,
> >
> > Seems it is clustered environment and other node(s) are being
> > disconnected, from the logs. Can you check the management host of
> > connected agents in the KVM host using net stat and the host value
> > in "/etc/cloudstack/agent/agent.properties".
> >
> > Also, check the ownership of the host in mgmt_server_id column of
> > the host table and validate it against the management server details
> > in
> mshost table.
> >
> > Thanks,
> > Suresh
> >
> > On Mon, Mar 16, 2020 at 4:37 PM Adam Witwicki
> > <awitwicki@oakfordis.com>
> > wrote:
> >
> > > Hi Guys,
> > >
> > > I need some help,
> > >
> > > One of my KVM agents is stuck in alert state - no amount of
> > > restarting the agent helps
> > >
> > >
> > >
> > > Cloudstack version 4.11.0.0
> > >
> > > Logs
> > > Management https://pastebin.com/JY2bvqmM Agent
> > > https://pastebin.com/G7g8wXNN
> > >
> > > All the other agents connect fine
> > >
> > > Thanks
> > >
> > > Adam
> > >
> > >
> > >
> > > Disclaimer Notice:
> > > This email has been sent by Oakford Technology Limited, while we
> > > have checked this e-mail and any attachments for viruses, we can
> > > not guarantee that they are virus-free. You must therefore take
> > > full responsibility for virus checking.
> > > This message and any attachments are confidential and should only
> > > be read by those to whom they are addressed. If you are not the
> > > intended recipient, please contact us, delete the message from
> > > your computer and destroy any copies. Any distribution or copying
> > > without our prior permission is prohibited.
> > > Internet communications are not always secure and therefore
> > > Oakford Technology Limited does not accept legal responsibility
> > > for this
> message.
> > > The recipient is responsible for verifying its authenticity before
> > > acting on the contents. Any views or opinions presented are solely
> > > those of the author and do not necessarily represent those of
> > > Oakford
> > Technology Limited.
> > > Registered address: Oakford Technology Limited, The Manor House,
> > > Potterne, Wiltshire. SN10 5PN.
> > > Registered in England and Wales No. 5971519
> > >
> > >
> > Disclaimer Notice:
> > This email has been sent by Oakford Technology Limited, while we
> > have checked this e-mail and any attachments for viruses, we can not
> > guarantee that they are virus-free. You must therefore take full
> > responsibility for virus checking.
> > This message and any attachments are confidential and should only be
> > read by those to whom they are addressed. If you are not the
> > intended recipient, please contact us, delete the message from your
> > computer and destroy any copies. Any distribution or copying without
> > our prior permission is prohibited.
> > Internet communications are not always secure and therefore Oakford
> > Technology Limited does not accept legal responsibility for this message.
> > The recipient is responsible for verifying its authenticity before
> > acting on the contents. Any views or opinions presented are solely
> > those of the author and do not necessarily represent those of
> > Oakford
> Technology Limited.
> > Registered address: Oakford Technology Limited, The Manor House,
> > Potterne, Wiltshire. SN10 5PN.
> > Registered in England and Wales No. 5971519
> >
> >
> Disclaimer Notice:
> This email has been sent by Oakford Technology Limited, while we have
> checked this e-mail and any attachments for viruses, we can not
> guarantee that they are virus-free. You must therefore take full
> responsibility for virus checking.
> This message and any attachments are confidential and should only be
> read by those to whom they are addressed. If you are not the intended
> recipient, please contact us, delete the message from your computer
> and destroy any copies. Any distribution or copying without our prior
> permission is prohibited.
> Internet communications are not always secure and therefore Oakford
> Technology Limited does not accept legal responsibility for this message.
> The recipient is responsible for verifying its authenticity before
> acting on the contents. Any views or opinions presented are solely
> those of the author and do not necessarily represent those of Oakford Technology Limited.
> Registered address: Oakford Technology Limited, The Manor House,
> Potterne, Wiltshire. SN10 5PN.
> Registered in England and Wales No. 5971519
>
>

--

Andrija Panić
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail
and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore
take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom
they are addressed. If you are not the intended recipient, please contact us, delete the message
from your computer and destroy any copies. Any distribution or copying without our prior permission
is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does
not accept legal responsibility for this message. The recipient is responsible for verifying
its authenticity before acting on the contents. Any views or opinions presented are solely
those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, Wiltshire. SN10
5PN.
Registered in England and Wales No. 5971519

Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail
and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore
take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom
they are addressed. If you are not the intended recipient, please contact us, delete the message
from your computer and destroy any copies. Any distribution or copying without our prior permission
is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does
not accept legal responsibility for this message. The recipient is responsible for verifying
its authenticity before acting on the contents. Any views or opinions presented are solely
those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, Wiltshire. SN10
5PN.
Registered in England and Wales No. 5971519

Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail
and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore
take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom
they are addressed. If you are not the intended recipient, please contact us, delete the message
from your computer and destroy any copies. Any distribution or copying without our prior permission
is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does
not accept legal responsibility for this message. The recipient is responsible for verifying
its authenticity before acting on the contents. Any views or opinions presented are solely
those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, The Manor House, Potterne, Wiltshire. SN10
5PN.
Registered in England and Wales No. 5971519


Mime
View raw message