Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <515C721E.4070908@widodh.nl>
Date: Wed, 03 Apr 2013 20:17:02 +0200
From: Wido den Hollander <wido@widodh.nl>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130308 Thunderbird/17.0.4
MIME-Version: 1.0
To: dev@cloudstack.apache.org
Subject: Re: [ACS41] Management server closes connection on port 8250
References: <515B2448.4010803@widodh.nl>
 <CA+2rt41bGLzag-3Di724Xq_LTtOmgkELc5SJq-+MpnzED3t=9Q@mail.gmail.com>
 <CA+2rt41iEhOvD1B6LMOLA2NRbbnmbEumMfqhwKNLDwFL5BBrWw@mail.gmail.com>
 <515B3B34.20800@widodh.nl>
 <20130402201235.GW40175@USLT-205755.sungardas.corp>
 <CA+2rt40yynm0kMtVJshEZtfSh1xCYNoxisfqnuCa5jXcJqO2dQ@mail.gmail.com>
 <515C6FB6.3090403@widodh.nl>
 <CALFpzo5Uk6YD8CoCqpd=ynQGExx9skv9eH7SN+9FekFsxMA7tw@mail.gmail.com>
In-Reply-To: 
 <CALFpzo5Uk6YD8CoCqpd=ynQGExx9skv9eH7SN+9FekFsxMA7tw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


On 04/03/2013 08:14 PM, Marcus Sorensen wrote:
> did you copy the key file from the old install?
>

Yes, I copied "key" which only contained "password".

Still not sure about it though. What I did find out, we need to do a 
trim on the contents of the key file, since a small whitespace at the 
end will already break your setup.

Wido

>
> On Wed, Apr 3, 2013 at 12:06 PM, Wido den Hollander <wido@widodh.nl> wrote:
>
>>
>>
>> On 04/03/2013 07:44 PM, Sheng Yang wrote:
>>
>>> On Tue, Apr 2, 2013 at 1:12 PM, Chip Childers <chip.childers@sungard.com>
>>> **wrote:
>>>
>>>   On Tue, Apr 02, 2013 at 10:10:28PM +0200, Wido den Hollander wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On 04/02/2013 08:50 PM, Sheng Yang wrote:
>>>>>
>>>>>> On Tue, Apr 2, 2013 at 11:49 AM, Sheng Yang <sheng@yasker.org> wrote:
>>>>>>
>>>>>>> On Tue, Apr 2, 2013 at 11:32 AM, Wido den Hollander <wido@widodh.nl>
>>>>>>>
>>>>>> wrote:
>>>>
>>>>> Hi,
>>>>>>>>
>>>>>>>> Since I've upgraded my management server to 4.1 my agents refuse to
>>>>>>>>
>>>>>>> connect,
>>>>
>>>>> their log says:
>>>>>>>>
>>>>>>>> 2013-04-02 20:26:11,207 INFO  [utils.nio.NioClient]
>>>>>>>>
>>>>>>> (Agent-Selector:null)
>>>>
>>>>> Connecting to 31.25.X.X:8250
>>>>>>>> 2013-04-02 20:26:11,209 ERROR [utils.nio.NioConnection]
>>>>>>>> (Agent-Selector:null) Unable to initialize the threads.
>>>>>>>> java.io.IOException: SSL: Fail to init SSL! java.io.IOException:
>>>>>>>>
>>>>>>> Connection
>>>>
>>>>> closed with -1 on reading size.
>>>>>>>>           at com.cloud.utils.nio.NioClient.**init(NioClient.java:83)
>>>>>>>>           at
>>>>>>>>
>>>>>>> com.cloud.utils.nio.**NioConnection.run(**NioConnection.java:108)
>>>>
>>>>>           at java.lang.Thread.run(Thread.**java:679)
>>>>>>>>
>>>>>>>> So I tried a telnet connection:
>>>>>>>>
>>>>>>>> Connection closed by foreign host.
>>>>>>>> root@stack01:~# telnet 31.25.X.X 8250
>>>>>>>> Trying 31.25.X.X...
>>>>>>>> Connected to 31.X.X.X.
>>>>>>>> Escape character is '^]'.
>>>>>>>> Connection closed by foreign host.
>>>>>>>> root@stack01:~#
>>>>>>>>
>>>>>>>> So I didn't close the connection, but it was the management server.
>>>>>>>>
>>>>>>>> I cranked up the logging to DEBUG, but nothing shows in any of the
>>>>>>>>
>>>>>>> logs, so
>>>>
>>>>> I have no clue why this isn't working.
>>>>>>>>
>>>>>>>> On the mgmt server I see Java in LISTEN state on port 8250
>>>>>>>>
>>>>>>>> There is no firewall on the management server (it's my lab!).
>>>>>>>>
>>>>>>>> Any clues to what this could be before I start filing a in Jira?
>>>>>>>>
>>>>>>> Since I'm
>>>>
>>>>> not sure if this is a bug.
>>>>>>>>
>>>>>>>
>>>>>>> Have you upgrade your agent?
>>>>>>>
>>>>>>
>>>>>>
>>>>> Yes, the Agent is also 4.1
>>>>>
>>>>>   Also you could try to enable TRACE for com.cloud.utils.nio to see more
>>>>>> log of NIO.
>>>>>>
>>>>>>
>>>>> I think you mean at the agent?
>>>>>
>>>>> But what I think is weird is that the mgmt server directly closes
>>>>> the telnet connection as well and nothing shows up in the logs.
>>>>>
>>>>> In the mgmt server log I found this:
>>>>>
>>>>> 2013-04-02 22:08:41,156 INFO  [utils.nio.NioServer]
>>>>> (AgentManager-Selector:null) NioConnection started and listening on
>>>>> 0.0.0.0/0.0.0.0:8250
>>>>>
>>>>> What I did notice is that when I shutdown the mgmt server the Agent
>>>>> connects for a second:
>>>>>
>>>>> 2013-04-02 22:07:53,689 INFO  [utils.nio.NioClient]
>>>>> (Agent-Selector:null) Connecting to 31.25.X.X:8250
>>>>> 2013-04-02 22:07:53,934 INFO  [utils.nio.NioClient]
>>>>> (Agent-Selector:null) SSL: Handshake done
>>>>>
>>>>> And then the mgmt server exists and the connection gets lost.
>>>>>
>>>>> ANY clues about this weird behavior?
>>>>>
>>>>> Wido
>>>>>
>>>>>   --Sheng
>>>>>>
>>>>>>>
>>>>>>> --Sheng
>>>>>>>
>>>>>>>>
>>>>>>>> Wido
>>>>>>>>
>>>>>>>
>>>>>
>>>> Wido - Can you please open a blocker bug to track this?  It may end up
>>>> being a non-issue or something...  but I want to know that we have it
>>>> outstanding when considering when to cut the RC.
>>>>
>>>>
>>> It remind me of one weird bug. The corrupted ssl keystore(due to some
>>> encryption/decryption of db which I don't understand well, I remember
>>> Kelven knows more about it) on mgmt server make it impossible to connect
>>> to
>>> mgmt server.
>>>
>>> Could you clear "ssl.keystore" entry in configuration table of db, as well
>>> as cloud.keystore file in the configuration path and restart the mgmt
>>> server and try again?
>>>
>>>
>> I "fixed" it about 10 minutes prior to your e-mail. It seems related to
>> CLOUDSTACK-1877: https://issues.apache.org/**jira/browse/CLOUDSTACK-1877<https://issues.apache.org/jira/browse/CLOUDSTACK-1877>
>>
>> The management server read the wrong db.properties and after fixing that
>> it worked.
>>
>> It seems to be due to some encryption issues because I had to completely
>> turn of db encryption in my db.properties.
>>
>> My mgmt server is now running, but not sure yet what the root cause is.
>>
>> Wido
>>
>>   --Sheng
>>>
>>>
>