cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Bouige <n.bou...@dimsi.fr>
Subject RE: certificate issue second mgmt-server
Date Fri, 04 May 2018 13:33:43 GMT
Hi All,


So, i was working on my primary issue with the certificate on my second node...i've stopped
and restart the first node and now the cloudstack-management is in status failed :'(


I got an "invalid specified classpath" and "Cannot parse command line arguments".

I modified the file cloudstack-management.service" to manually set up these varirable $JAR:$CLASSPATH.


After a deamon-reload and a restart of the service, it's  run around 30 sec and fail again,
here the log from the status service:


mai 04 20:43:24 ASPRCSMGMT01 systemd[1]: Starting CloudStack Management Server...
mai 04 20:43:24 ASPRCSMGMT01 systemd[1]: Started CloudStack Management Server.
mai 04 20:44:00 ASPRCSMGMT01 sudo[5033]:    cloud : TTY=unknown ; PWD=/var/log/cloudstack/management
; USER=root ; COMMAND=/bin/mkdir -p /systemvm_mnt
mai 04 20:44:00 ASPRCSMGMT01 sudo[5036]:    cloud : TTY=unknown ; PWD=/var/log/cloudstack/management
; USER=root ; COMMAND=/bin/mount -o loop /usr/share/cloudstack-common/vms/systemvm.iso /systemvm_mnt
mai 04 20:44:00 ASPRCSMGMT01 sudo[5041]:    cloud : TTY=unknown ; PWD=/var/log/cloudstack/management
; USER=root ; COMMAND=/bin/umount /systemvm_mnt
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: cloudstack-management.service: main process exited,
code=exited, status=1/FAILURE
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: cloudstack-management.service: control process exited,
code=exited status=255
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: Unit cloudstack-management.service entered failed
state.
mai 04 20:44:15 ASPRCSMGMT01 systemd[1]: cloudstack-management.service failed.

>From /var/log/messages :

May  4 20:43:28 ASPRCSMGMT01 systemd: Configuration file /usr/lib/systemd/system/cloudstack-management.service
is marked executable. Please remove executable permission bits. Proceeding anyway.
May  4 20:44:15 ASPRCSMGMT01 systemd: cloudstack-management.service: main process exited,
code=exited, status=1/FAILURE
May  4 20:44:15 ASPRCSMGMT01 systemd: cloudstack-management.service: control process exited,
code=exited status=255
May  4 20:44:15 ASPRCSMGMT01 systemd: Unit cloudstack-management.service entered failed state.



This management was working fine for one month and i restarted it without any problem before...


If one of you have any idea, it would be appreciated,

Thanks upfront for any help,

Best regards,

Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d’Asnière – 92600 Asnière sur Seine
T/ +33 (0)6 28 98 53 40


________________________________
De : Nicolas Bouige <n.bouige@dimsi.fr>
Envoyé : mercredi 2 mai 2018 10:54:36
À : users@cloudstack.apache.org
Objet : RE: certificate issue second mgmt-server

Rohit,

Thanks for the details, i'll keep you update if its work.

Best regards,
N.B

-----Message d'origine-----
De : Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Envoyé : mercredi 2 mai 2018 10:39
À : users@cloudstack.apache.org
Objet : Re: certificate issue second mgmt-server

Nicolas,

Yes, if you've existing systemvms and KVM hosts changing the ca private/public key could cause
system-wide cert issue. You can retry shutting down both management server(s), start the primary
mgmt server to come up first and then start/deploy other mgmt servers one by one.


- Rohit

<https://cloudstack.apache.org>



________________________________
From: Nicolas Bouige <n.bouige@dimsi.fr>
Sent: Wednesday, May 2, 2018 1:57:07 PM
To: users@cloudstack.apache.org
Subject: RE: certificate issue second mgmt-server

Hi Rohit,

Thanks for your answer, i can't remember if i added the second node before the end of the
initialization maybe i was too impatient :/ I'll give a try this week with your workaround.
Your workaround will affect also KVM server and System-VM ?

So, i guess, it's not enought to delete the second node and redeploy it ?

Best regards,
N.B

rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




-----Message d'origine-----
De : Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Envoyé : mardi 1 mai 2018 12:56
À : users@cloudstack.apache.org
Objet : Re: certificate issue second mgmt-server

Hi Nicolas,


Did you deploy multiple managements at the same time? When you deploy multiple management
server(s), wait for the first management server to initialize database where it sets up some
default offerings, global settings and the root CA keypair and certificate. Only when you
see the first management server's UI in browser, proceed with deployment of other management
server(s).


For your environment, you can test this workaround and let me know if that works for you:


  1.  Shutdown all the management server(s).
  2.  Delete ca keypair and cert:
 delete from configuration where name like "ca.plugin.root.private.key";  delete from configuration
where name like "ca.plugin.root.public.key";  delete from configuration where name="ca.plugin.root.ca.certificate";
  3.  Start one management server and wait for it to complete internal setup, until you see
the UI.
  4.  Start all the other management server(s).



- Rohit

<https://cloudstack.apache.org>



________________________________
From: Nicolas Bouige <n.bouige@dimsi.fr>
Sent: Monday, April 30, 2018 2:59:29 PM
To: users@cloudstack.apache.org
Subject: certificate issue second mgmt-server

Hello All,


I have an issue with one of my Cloudstack mgmt-server (4.11)

The second node has been deployed with the command  "cloudstack-setup-databases cloud:dbpassword@dbhost"


i didnt have any problem during few days and now sometimes i got an error on web GUI when
i perfom some basic task, the error is "Resource [Host:1] is unreachable: Host 1: Unable to
reach the peer that the agent is connected"


After a quick investigation, i had to stop cloudstack-management service from second mgmt-server
and i noticed a lot of messages related with ca-certificate used by cloudstack :


2018-04-27 11:18:24,076 ERROR [c.c.u.n.Link] (StatsCollector-1:ctx-82335701) (logid:95fda6d7)
SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local
address=/172.16.22.61:60128, remote address=/172.16.22.60:8250. The client may have invalid
ca-certificates.
2018-04-27 11:18:24,076 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:ctx-82335701)
(logid:95fda6d7) Unable to connect to peer management server: 130719784044197, ip: 172.16.22.60
due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management
server '130719784044197' on 172.16.22.60:8250
java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with
peer management server '130719784044197' on 172.16.22.60:8250
        at com.cloud.agent.manager.ClusteredAgentManagerImpl.connectToPeer(ClusteredAgentManagerImpl.java:529)
        at com.cloud.agent.manager.ClusteredAgentAttache.send(ClusteredAgentAttache.java:177)
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:456)
        at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
        at com.cloud.agent.manager.AgentManagerImpl.easySend(AgentManagerImpl.java:954)
        at com.cloud.resource.ResourceManagerImpl.getHostStatistics(ResourceManagerImpl.java:2645)
        at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy178.getHostStatistics(Unknown Source)
        at com.cloud.server.StatsCollector$HostCollector.runInContext(StatsCollector.java:438)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-04-27 11:18:24,077 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-1:ctx-82335701)
(logid:95fda6d7) Seq 9-9075597674081682614: Unable to forward null
2018-04-27 11:18:24,177 ERROR [c.c.u.n.Link] (StatsCollector-1:ctx-82335701) (logid:95fda6d7)
SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local
address=/172.16.22.61:60130, remote address=/172.16.22.60:8250. The client may have invalid
ca-certificates.
2018-04-27 11:18:24,177 WARN  [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-1:ctx-82335701)
(logid:95fda6d7) Unable to connect to peer management server: 130719784044197, ip: 172.16.22.60
due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management
server '130719784044197' on 172.16.22.60:8250

Im not familiar with the using of self-signed certificate in cloudstack, do you know where
i can find out more information to investigate deeper ? or if you have any idea ?
I tried to check keystore on both mgmt-server but i need a password i havnt...


Thanks upfront,
Have a nice day,

Best regards,

Nicolas Bouige
DIMSI
cloud.dimsi.fr<http://www.cloud.dimsi.fr>
4, avenue Laurent Cely
Tour d'Asnière - 92600 Asnière sur Seine

T/ +33 (0)6 28 98 53 40


rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message