cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kristian Liivak <k...@wavecom.ee>
Subject Re: System VM Issues 4.11?
Date Mon, 26 Feb 2018 13:46:14 GMT
Rohit,

Systemvms are 4.11, i can see that in console before login :)
cat /proc/cmdline returnes some info. Please see attached pics

Management sevrer is working just fine. I dont have however autostart on in upgrade time.
Im using old centos7 management server. Once allready made upgrade from ACS 4.9 to 4.10
Now from 4.10 to 4.11..

Is there maybe point to use clean 4.11 management server to resolve system vm issue ?
And do i need to have HVM flag cheked with 4.11 system management template install ?
There is nothing about it in  instructions. However i have tried both ways without any success.


Lugupidamisega / Regards
 
Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: kris@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee

----- Original Message -----
From: "rohit yadav" <rohit.yadav@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Sunday, February 25, 2018 4:04:54 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the cmdline options
from XenServer to patch itself, reconfigure the nics. The ifconfig does not return at least
the control nic. Can you check what `cat /etc/cloudstack-release` returns on the systemvm
(this is to test that the systemvm template is indeed the 4.11 one).


On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me that the mgmt
server is failing to run check for cloudstack-management start failures, see /etc/default/cloudstack-management
and the cloudstack-management.service to find why the process fails, perhaps the classpath
(or JAVA_HOME/java path) needs to be fixed?


Lastly, can you also comment if you're using a new host/server to run mgmt server, or you
did an in-place upgrade?


- Rohit

<https://cloudstack.apache.org>



________________________________
From: Kristian Liivak <kris@wavecom.ee>
Sent: Saturday, February 24, 2018 8:31:35 AM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

I managed fix absent of hosts. It was host tags related afterall. db host_tags contained tags
before xenserver update in past. After dropping them update succeeded until new systemvm deployment.
Like  others said in thread, systemvm are successfully deployd, but not network will be started
and they will not come up.
I gathered logs , maybe you have time to check.
https://wavecom.ee/files/cloudstack.rar


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: kris@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.yadav@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

----- Original Message -----
From: "rohit yadav" <rohit.yadav@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Wednesday, February 21, 2018 6:25:11 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Do you get any results if you execute the listRouters/listHosts apis using say a CLI such
as cloudmonkey as the root admin?


🐵 > list routers listall=true


🐵 > list hosts

Also, do you have any modifications to the UI, or webserver/middleware? I looked at the logs,
and I could not find any exceptions or anything suspicious.

About host tags, clearing the tags and doing reconnect causes cloudstack to scp/copy scripts
and files to xenserver host which usually happens when you upgrade cloudstack. It's not an
issue for you as in your logs I see such activities after cloudstack upgrades. (otherwise,
it's done using, something like:

# for host in $(xe host-list | grep ^uuid | awk '{print $NF}') ; do xe host-param-clear uuid=$host
param-name=tags; done; )

- Rohit

<https://cloudstack.apache.org>



________________________________
From: Kristian Liivak <kris@wavecom.ee>
Sent: Tuesday, February 20, 2018 10:25:42 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

Made another shot.

Recreated all systemvm before update and get rid most of errors had before.

Like before no host nor systemvm was present in gui
However i saw correct numbers under infrastructure.
I attached some pics form gui  host and systemvm  selection.

I did destroyd old systemvms last time
New ones started without network.
New templates were present and new systemvm are created from 4.11 templates, checked from
xen console.

My management server resides in different vmware enviroment, however it have same management
network like all my xenservers.

I started with 4.9, where dynamic roles  were allready introduced, but i doublechecked.

My old enviroment was configured with different port using custom ssl, so therefore no cache.

Enviroment is only xenserver7.0 hosts

Uploaded new management logs and also /var/log/messages where update logs are present https://wavecom.ee/files/logs.rar

Maybe you can find time to check..

And i can allways rollback management where i was to have additional info

PS what you mean about : reset host flags and reconnect on all such hosts. i tried add hosts
agan in gui bot that ended with failure.

Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: kris@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



----- Original Message -----
From: "rohit yadav" <rohit.yadav@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Sunday, February 18, 2018 3:52:00 PM
Subject: Re: System VM Issues 4.11?

Thanks Kristian,


Please confirm:

- Did you see the new 4.11 systemvmtemplate and tried to destroy old systemvms and/or virtual
routers?

- Are all the xenserver hosts version 7.0 in all your clusters?

- Was the upgrade in-place (on the same host), or on a different server?

- Is dynamic roles enabled or you're still using the old/deprecated static roles i.e. commands.properties
file?

- Did you try accessing the UI in incognito mode or after clearing the browser cache?


Looking at the logs, some errors:


>  2018-01-01 00:01:22,481 ERROR [c.c.u.s.SshHelper] (DirectAgent-29:ctx-62676ad7) (logid:48c55dee)
SSH execution of command /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.3.20 -g has an
error status code in return. Result output: /root/func.sh: line 61: /tmp/1514746614947437999-28844-biglock.lock:
Read-only file system

> 2018-01-01 00:01:22,482 WARN  [c.c.h.x.r.w.x.XenServer56NetworkUsageCommandWrapper] (DirectAgent-29:ctx-62676ad7)
(logid:48c55dee) Failed to get network usage


This looks like a case where an old (pre 4.11) or new (4.11) based virtual router ended up
having read-only file system. If there are old virtual routers they need to be removed, restart
network with cleanup=true.


> 2018-01-01 00:01:24,503 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager
Timer:ctx-198e0371) (logid:509434bb) Unable to create master connection to host(172.16.10.13)
, due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect timed
out                          > 2018-01-01 00:01:24,503 DEBUG [c.c.h.Status] (ClusteredAgentManager
Timer:ctx-198e0371) (logid:509434bb) Transition:[Resource state = Enabled, Agent event = AgentDisconnected,
Host id = 864, name = xen-13-10-b1-khi02]

I'm not sure why your mgmt server was unable to make a connection with existing xenserver
host. After a major version upgrade, on reconnection scripts will be copied to your xenserver
host. In case that failed (because I don't have your upgrade logs), you'll need to reset host
flags and reconnect on all such hosts.

> 2018-01-01 00:01:26,367 ERROR [c.c.u.s.SshHelper] (AgentTaskPool-12:ctx-cfea21e4) (logid:c2b516fe)
SSH execution of command xe sm-list | grep "resigning of duplicates" has an error status code
in return. Result output:


Not sure about these ones.


> 2018-01-01 00:40:09,411 WARN  [c.c.h.x.r.CitrixResourceBase] (DirectAgent-271:ctx-c9e202e1)
(logid:1f6aafe2) Unable to start i-1744-15758-VM due to-

com.cloud.utils.exception.CloudRuntimeException: createIsoSRbyURI failed! mountpoint: 172.16.2.37/nfs/KHI03/template/tmpl/1326/465
due to SR_BACKEND_FAILURE_222Could not mount the directory specified in Device Configuration
>---at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createIsoSRbyURI(CitrixResourceBase.java:988)

Can you check if your storage pools are accessible to your hosts?


Lastly, if you attempt upgrade again (or perhaps on a test environment), do keep the logs
for future reference for example sharing with us in case it fails for you again.



- Rohit

<https://cloudstack.apache.org>



________________________________
From: Kristian Liivak <kris@wavecom.ee>
Sent: Saturday, February 17, 2018 8:49:26 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit

Here it is https://wavecom.ee/files/management-server.rar
Log beginning little bit earlier then upgrade.
I did,nt fallow errors during gui access this time.
If it helps i can allways revert management server where i was in that moment and grab more
info.

Lugupidamisega / Regards

Kristian Liivak

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: kris@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



----- Original Message -----
From: "rohit yadav" <rohit.yadav@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Saturday, February 17, 2018 9:15:56 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Yes, can you share the logs and if you saw any sql/db or other errors when you tried to access
the UI?


- Rohit

<https://cloudstack.apache.org>



________________________________
From: Kristian Liivak <kris@wavecom.ee>
Sent: Saturday, February 17, 2018 3:07:04 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi All

I also upgraded from 4.10->4.11 and had similar issues ,even more.
After upgrade, Gui under infrastructure i didnt show any host nor systemvm anymore , except
VR
Destroyd consoleproxy, new was created but no network running.

Enviroment is Xenserver7 and i also runned before upgrade sql update mentioned below .
Before reverting back i grabbed all log files. If anyone is intrested i can make them downloadable.


Lugupidamisega / Regards

Kristian Liivak

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: kris@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



----- Original Message -----
From: "Paul Angus" <paul.angus@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Friday, February 16, 2018 10:45:26 AM
Subject: RE: System VM Issues 4.11?

Sami,

Have you run the update step which is mentioned here?
http://markmail.org/message/f42kqr3mx4r4hgih

I don't KNOW if it is causing your problem, but if not, it won't help if it is missing.


Kind regards,

Paul Angus

paul.angus@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-----Original Message-----
From: sami [mailto:sami.rajala@fapps.com]
Sent: 16 February 2018 07:44
To: users@cloudstack.apache.org
Subject: Fwd: System VM Issues 4.11?


Hi,

Actually I have a related issue, The System VMs stay in starting state.
when I look situation on XenCenter, I see CPVM /SSVM running, but does not deploy configuration
(no linklocal/private/publi IP on those VM¹s)

I did not try VPC router yet, The Network routers starting ok.


ACS upgraded 4.10 to 4.11 on Centos 6
xenserver 7.0
Advance network

Regards
-Sami


> 15.2.2018 20.14, "Rohit Yadav" <rohit.yadav@shapeblue.com>:
>
>> Hi Benjamin,
>>
>> Starting 4.11, systemvms don't reboot after the patching operation so
>> the shared observation matches expectations. Do you see vpc router
>> vms stuck in starting state, can you share the logs and details of
>> your hypervisor and other environment details?
>>
>>
>> - Rohit
>>
>> <https://cloudstack.apache.org>
>>
>>
>>
>> ________________________________
>> From: Benjamin Naber <benjamin.naber@coders-area.de>
>> Sent: Thursday, February 15, 2018 6:01:51 PM
>> To: users
>> Subject: System VM Issues 4.11?
>>
>> Hi @all,
>>
>> I've installed 4.11 from shape blue CentOS 7 Repository.
>> System VM and Router VM Upgrades working well. But have some issues
>> if I deploy a new Vpc. Router VMs stalled in state starting.
>> Inspecting the hypervisor shows that the router VM doesn't reboot and
>> doesn't deploy the configuration. Anyone other have the same issues?
>>
>> Kind Regards
>>
>> Benjamin
>>
>> Von meinem Huawei-Mobiltelefon gesendet
>>
>> rohit.yadav@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>>
>>
Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message