cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From martin kolly <martin.ko...@senselan.ch>
Subject Re: CS 4.9 NIO Selector wait time PR-1601
Date Thu, 25 Aug 2016 10:04:17 GMT

thanks for your reply.

This morning we repeated the upgrade process from 4.8 to 4.9 with the
following repository:
http://packages.shapeblue.com/cloudstack/upstream/debian/4.9/.
<http://packages.shapeblue.com/cloudstack/upstream/debian/4.9/>
Unfortunately we run into the same issue:

/2016-08-25 09:49:00,660 INFO  [utils.nio.NioClient] (main:null)
(logid:) Connecting to 10.100.12.10:8250//
//2016-08-25 09:49:00,668 WARN  [utils.nio.Link] (main:null) (logid:)
This SSL engine was forced to close inbound due to end of stream.//
//2016-08-25 09:49:00,668 ERROR [utils.nio.NioClient] (main:null)
(logid:) SSL Handshake failed while connecting to host: 10.100.12.10
port: 8250//
//2016-08-25 09:49:00,668 ERROR [utils.nio.NioConnection] (main:null)
(logid:) Unable to initialize the threads.//
//java.io.IOException: SSL Handshake failed while connecting to host:
10.100.12.10 port: 8250//
//    at com.cloud.utils.nio.NioClient.init(NioClient.java:67)//
//    at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)//
//    at com.cloud.agent.Agent.start(Agent.java:237)//
//    at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)//
//    at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)//
//    at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)//
//    at com.cloud.agent.AgentShell.start(AgentShell.java:456)//
//    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)//
//    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)//
//    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)//
//    at java.lang.reflect.Method.invoke(Method.java:606)//
//    at
org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)//
//2016-08-25 09:49:00,669 INFO  [utils.exception.CSExceptionErrorCode]
(main:null) (logid:) Could not find exception:
com.cloud.utils.exception.NioConnectionException in error code list for
exceptions//
//2016-08-25 09:49:00,669 WARN  [cloud.agent.Agent] (main:null) (logid:)
NIO Connection Exception 
com.cloud.utils.exception.NioConnectionException: SSL Handshake failed
while connecting to host: 10.100.12.10 port: 8250//
//2016-08-25 09:49:00,670 INFO  [cloud.agent.Agent] (main:null) (logid:)
Attempted to connect to the server, but received an unexpected
exception, trying again.../

*KVM Hosts:
*/# java -version
java version "1.7.0_95"
OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-0ubuntu0.14.04.1)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

# dpkg --get-selections | grep -e 'jdk' -e 'java'
ca-certificates-java                install
java-common                    install
libcommons-daemon-java                install
openjdk-7-jre-headless:amd64            install
tzdata-java                    install

# apt-cache policy cloudstack-agent
cloudstack-agent:
  Installed: 4.9.0
  Candidate: 4.9.0
  Version table:
 *** 4.9.0 0
        500
http://packages.shapeblue.com/cloudstack/upstream/debian/4.9/ ./ Packages
        100 /var/lib/dpkg/status

# find /usr/share/ -name "cloud-utils*.jar"
/usr/share/cloudstack-agent/lib/cloud-utils-4.9.0.jar
# md5sum /usr/share/cloudstack-agent/lib/cloud-utils-4.9.0.jar
a8de7306d7c80b5a73e93b83afdd119f 
/usr/share/cloudstack-agent/lib/cloud-utils-4.9.0.jar


/*Management Server:
*/# java -version//
//java version "1.7.0_95"//
//OpenJDK Runtime Environment (IcedTea 2.6.4)
(7u95-2.6.4-0ubuntu0.14.04.1)//
//OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)//
//
//# dpkg --get-selections | grep -e 'jdk' -e 'java'//
//ca-certificates-java                install//
//java-common                    install//
//libcommons-collections3-java            install//
//libcommons-daemon-java                install//
//libcommons-dbcp-java                install//
//libcommons-pool-java                install//
//libecj-java                    install//
//libgeronimo-jta-1.1-spec-java            install//
//libmysql-java                    install//
//libservlet2.5-java                install//
//libtomcat6-java                    install//
//openjdk-7-jre-headless:amd64            install//
//tzdata-java                    install//
//
//# apt-cache policy cloudstack-management//
//cloudstack-management://
//  Installed: 4.9.0//
//  Candidate: 4.9.0//
//  Version table://
// *** 4.9.0 0//
//        500
http://packages.shapeblue.com/cloudstack/upstream/debian/4.9/ ./ Packages//
//        100 /var/lib/dpkg/status///

/# find /usr/share/ -name "cloud-utils*.jar"//
///usr/share/cloudstack-management/webapps/client/WEB-INF/lib/cloud-utils-4.9.0.jar//
///usr/share/cloudstack-agent/lib/cloud-utils-4.9.0.jar//
///usr/share/cloudstack-usage/lib/cloud-utils-4.9.0.jar//
//# md5sum
/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/cloud-utils-4.9.0.jar//
//a8de7306d7c80b5a73e93b83afdd119f 
/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/cloud-utils-4.9.0.jar//
//# md5sum /usr/share/cloudstack-agent/lib/cloud-utils-4.9.0.jar//
//a8de7306d7c80b5a73e93b83afdd119f 
/usr/share/cloudstack-agent/lib/cloud-utils-4.9.0.jar//
//# md5sum /usr/share/cloudstack-usage/lib/cloud-utils-4.9.0.jar//
//a8de7306d7c80b5a73e93b83afdd119f 
/usr/share/cloudstack-usage/lib/cloud-utils-4.9.0.jar/

The classpath.conf was not modified:
/# cat /etc/cloudstack/management/classpath.conf
#!/bin/bash
#...

SYSTEMJARS=""
SCP=$(build-classpath $SYSTEMJARS 2>/dev/null) ; if [ $? != 0 ] ; then
export SCP="" ; fi
MCP=""
DCP="/usr/share/tomcat6/bin/bootstrap.jar:/usr/share/tomcat6/bin/tomcat-juli.jar"
CLASSPATH=$SCP:$DCP:$MCP:/etc/cloudstack/management:/usr/share/cloudstack-management/setup
for jarfile in ""/* ; do
    if [ ! -e "$jarfile" ] ; then continue ; fi
    CLASSPATH=$jarfile:$CLASSPATH
done
for plugin in ""/* ; do
    if [ ! -e "$plugin" ] ; then continue ; fi
    CLASSPATH=$plugin:$CLASSPATH
done
for vendorconf in "/etc/cloudstack/management"/vendor/* ; do
    if [ ! -d "$vendorconf" ] ; then continue ; fi
    CLASSPATH=$vendorconf:$CLASSPATH
done
export CLASSPATH
if ([ -z "$JAVA_HOME" ] || [ ! -d "$JAVA_HOME" ]) && [ -d
/usr/lib/jvm/jre-1.7.0 ]; then
     export JAVA_HOME=/usr/lib/jvm/jre-1.7.0
fi
PATH=$JAVA_HOME/bin:/sbin:/usr/sbin:$PATH
export PATH/

Regards
Martin

On 08/24/2016 06:56 PM, Rohit Yadav wrote:
>
> Martin,
>
>
> Were you able to fix your issue after installing packages from the
> repo Will shared and restarting the services?
>
> I've not personally tested the apt-get.eu repo, but I had earlier
> built this repo which I'm personally using in my local KVM-trusty
> based cloud: http://packages.shapeblue.com/cloudstack/upstream/debian/4.9/
>
>
> If you're still getting the error, can you share the JRE version
> you're running, both on the mgmt server and on the KVM hosts? You can
> run java -version, or share output of "dpkg --get-selections | grep -e
> 'jdk' -e 'java'". Are you running CloudStack with any additional plugins?
>
> From the logs, looks like there are mixed jar files,
> NioConnectionException class was not found -- something's wrong with
> your installation. there must be a cloud-utils jar file make sure your
> installation don't have multiple copies/versions of jars
> (somewhere) in the in /usr/share/cloudstack-common and in
> /usr/share/cloudstack-management/webapps/client/ paths:
>
> Could not find exception:
> com.cloud.utils.exception.NioConnectionException in error code list for
> exceptions
> The error "Unable to initialize the threads." suggests, JVM was not
> able to spawn threads. I would like to know your JRE version and any
> other settings configured in /etc/cloudstack/management/classpath.conf
> (and there are bunch of other files where JAVA_OPTS might have been
> overridden). Note: For now you should only be using JRE1.7.
>
>
> Regards.
>
> rohit.yadav@shapeblue.com 
> www.shapeblue.com
> @shapeblue
>   
>
>   
>
> ------------------------------------------------------------------------
> *From:* martin kolly <martin.kolly@senselan.ch>
> *Sent:* 24 August 2016 19:53:26
> *To:* dev@cloudstack.apache.org; Rohit Yadav
> *Subject:* Re: CS 4.9 NIO Selector wait time PR-1601
>  
> Thanks Will!
>
> yes the repo is pointing to 4.9 release for all KVMs and for the
> management server:
> /cloudstack:~# cat /etc/apt/sources.list.d/cloudstack.list //
> //deb http://cloudstack.apt-get.eu/ubuntu trusty 4.9/
>
> All KVM agents and the mgmt server are upgraded to release 4.9 based
> on the documentation.We have restarted all the cloudstack-agents and
> the cloudstack-management service as well.
>
> Network traces are showing packets from KVM <-> Mgmt on port 8250.
> there is no security device in between.
>
> thanks
> fanfarlo
>
>
>
>
> On 08/24/2016 04:13 PM, Will Stevens wrote:
>> @rohit, I am guessing they should be installing the cloudstack-agent using
>> the following repo right?  That is what is described in the upgrade (trusty
>> instead of precise though).
>>
>> http://cloudstack.apt-get.eu/ubuntu/dists/trusty/4.9/
>>
>> @fanfarlo, are your repo's setup to point to the new 4.9 version?
>>
>> cheers,
>>
>> will
>>
>> On Wed, Aug 24, 2016 at 9:46 AM, Rohit Yadav <rohit.yadav@shapeblue.com>
>> wrote:
>>
>>> The PR and fix already exists in 4.9.0 release. Please make sure to
>>> upgrade all of your management server(s) and KVM agents and then also
>>> restart them after the upgrade.
>>>
>>>
>>> If you are seeing SSL handshake failures, it could be due to network or
>>> security issue and most likely due to mismatch between CloudStack mgmt
>>> server and KVM agent version.
>>>
>>>
>>> Regards.
>>>
>>> rohit.yadav@shapeblue.com
>>> www.shapeblue.com
>>> @shapeblue
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* Will Stevens <williamstevens@gmail.com>
>>> *Sent:* 24 August 2016 18:17:17
>>> *To:* dev@cloudstack.apache.org; Rohit Yadav
>>> *Subject:* Re: CS 4.9 NIO Selector wait time PR-1601
>>>
>>>
>>> That PR is already merged, so you don't have to do anything to get that
>>> code, you already have it.
>>>
>>> @rohit, can you review this?  I think this is a similar to the issue Simon
>>> reported earlier.
>>>
>>> Will
>>>
>>> On Aug 24, 2016 6:56 AM, "fanfarlo" <fanfarlo2@gmail.com> wrote:
>>>
>>>> hi all
>>>>
>>>> We have the following environment:
>>>> - OS: Debian 14.04 (hypervisors and management)
>>>> - 4 KVM Hosts
>>>> - Cloudstack Release 4.9 with local database
>>>>
>>>> Since we upgraded to Release 4.9 the KVM hosts no longer connect to the
>>>> management Server. Upgrade procedure was followed as described:
>>>> http://docs.cloudstack.apache.org/projects/cloudstack-releas
>>>> e-notes/en/4.9.0/upgrade/upgrade-4.8.html
>>>>
>>>>
>>>> On the KVM hosts we have the following error message:
>>>> /2016-08-24 10:42:49,678 INFO  [utils.exception.CSExceptionErrorCode]
>>>> (main:null) (logid:) Could not find exception:
>>>> com.cloud.utils.exception.NioConnectionException in error code list for
>>>> exceptions
>>>> 2016-08-24 10:42:49,678 WARN  [cloud.agent.Agent] (main:null) (logid:)
>>>> NIO Connection Exception
>>>> com.cloud.utils.exception.NioConnectionException: SSL Handshake failed
>>>> while connecting to host: 10.100.12.10 port: 8250
>>>> 2016-08-24 10:42:49,678 INFO  [cloud.agent.Agent] (main:null) (logid:)
>>>> Attempted to connect to the server, but received an unexpected
>>>> exception, trying again...
>>>> 2016-08-24 10:42:54,679 INFO  [utils.nio.NioClient] (main:null) (logid:)
>>>> Connecting to 10.100.12.10:8250
>>>> 2016-08-24 10:42:54,684 WARN  [utils.nio.Link] (main:null) (logid:) This
>>>> SSL engine was forced to close inbound due to end of stream.
>>>> 2016-08-24 10:42:54,684 ERROR [utils.nio.NioClient] (main:null) (logid:)
>>>> SSL Handshake failed while connecting to host: 10.100.12.10 port: 8250
>>>> 2016-08-24 10:42:54,685 ERROR [utils.nio.NioConnection] (main:null)
>>>> (logid:) Unable to initialize the threads.
>>>> java.io.IOException: SSL Handshake failed while connecting to host:
>>>> 10.100.12.10 port: 8250
>>>>     at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
>>>>     at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
>>>>     at com.cloud.agent.Agent.start(Agent.java:237)
>>>>     at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
>>>>     at
>>>> com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
>>>>     at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
>>>>     at com.cloud.agent.AgentShell.start(AgentShell.java:456)
>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>     at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:57)
>>>>     at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>>     at java.lang.reflect.Method.invoke(Method.java:606)
>>>>     at
>>>> org.apache.commons.daemon.support.DaemonLoader.start(DaemonL
>>>> oader.java:243)
>>>> 2016-08-24 10:42:54,685 INFO  [utils.exception.CSExceptionErrorCode]
>>>> (main:null) (logid:) Could not find exception:
>>>> com.cloud.utils.exception.NioConnectionException in error code list for
>>>> exceptions
>>>> 2016-08-24 10:42:54,685 WARN  [cloud.agent.Agent] (main:null) (logid:)
>>>> NIO Connection Exception
>>>> com.cloud.utils.exception.NioConnectionException: SSL Handshake failed
>>>> while connecting to host: 10.100.12.10 port: 8250
>>>> 2016-08-24 10:42:54,686 INFO  [cloud.agent.Agent] (main:null) (logid:)
>>>> Attempted to connect to the server, but received an unexpected
>>>> exception, trying again.../
>>>>
>>>>
>>>> Port is open on the management server, there is no firewall in between.
>>>> We found that there was a bug report here:
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9348. There is a PR
>>>> changing the NIO Selector wait time:
>>>> https://github.com/apache/cloudstack/pull/1601 which was merged into
>>>> master branch.
>>>>
>>>> Since we installed Release 4.9 we probably need to patch the
>>>> NioConection.class as described in PR1601 , right?
>>>>
>>>> kvm03# unzip -v /usr/share/cloudstack-agent/lib/cloud-utils-4.9.0.jar  |
>>>> grep NioConnection
>>>>     3923  Defl:N     1778  55% 2016-08-02 09:28 05aaf7d5
>>>> com/cloud/utils/nio/NioConnection$1.class
>>>>      881  Defl:N      495  44% 2016-08-02 09:28 e378984c
>>>> com/cloud/utils/nio/NioConnection$ChangeRequest.class
>>>>    15410  Defl:N     7130  54% 2016-08-02 09:28 b3281f5a
>>>> com/cloud/utils/nio/NioConnection.class
>>>>     1134  Defl:N      584  49% 2016-08-02 09:28 8d5cb4a8
>>>> com/cloud/utils/exception/NioConnectionException.class
>>>>
>>>> Due to a lack of java expertise we have some basic questions:
>>>> - Is there a patched jar file available ? public build server?
>>>> - Do we need to create the jar from sources ? procedure?
>>>> - How do we apply the patch ?
>>>>
>>>> many thanks!
>>>> fanfarlo
>>>>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message