Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Thu, 28 Jan 2016 06:32:58 -0800
Message-ID: <20160128063258.Horde.KcvAjxBxnkKg6xgqDL1fuMt@webmail.soleks.net>
From: Daniel Mezentsev <dan@soleks.com>
To: dev@cloudstack.apache.org
Subject: Re: Issue with VR after upgrade to 4.6.2
References: <20160123193221.Horde.6YLMLUKdrWkgQBfkCOtgux6@webmail.soleks.net>
 <56A4117B.7070404@apache.org>
 <AC4588BF-F5F5-4D25-AD0E-AE84B54CAD15@schubergphilis.com>
 <56A49CB9.5090801@apache.org> <56A49FCC.60002@apache.org>
 <20160125152854.Horde.TVzBZW_dJR5iu7yNHc9c_N-@webmail.soleks.net>
 <3ACFB5B0-1893-47F3-9FCF-8CC4F0724528@schubergphilis.com>
 <20160125204017.Horde.Yf-2Ln9CR8v6TuG6ikwmPpL@webmail.soleks.net>
 <0DA65B64-EC17-4189-AC35-54BD5743F6E6@schubergphilis.com>
 <20160128041751.Horde.p97sUkSCBwQ0J92zy41pXS-@webmail.soleks.net>
 <16225E5D-3A94-43E7-8624-9B79DA1D0D56@schubergphilis.com>
 <20160128055934.Horde.1nqPRHLEoU9e3git_eBljcD@webmail.soleks.net>
 <DCD51E6D-C0CF-4F13-B31B-DC00CE7E4097@schubergphilis.com>
In-Reply-To: <DCD51E6D-C0CF-4F13-B31B-DC00CE7E4097@schubergphilis.com>
User-Agent: Horde Application Framework 5
Content-Type: multipart/alternative; boundary="=_DRVECJvfNul2kNyzbPNFnlt"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

--=_DRVECJvfNul2kNyzbPNFnlt
Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes
Content-Description: Plaintext Message
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

  Increased settings to 60. Got VR(s) up and running.
Interesting.

> Hi Daniel,
>
> Can you please bump that setting to 15 or 20 and restart mgt server?
> Then start the router again.
>
> Let me know!
>
> Regards, Remi
>
> Sent from my iPhone
>
>> On 28 Jan 2016, at 07:00, Daniel Mezentsev <dan@soleks.com> wrote:
>>
>> Hi Remi,
>>
>> Setup is the following
>> Cloudstack 4.7.1
>> One cluster of XenServers (6.2) - 4 hosts
>> Advanced network
>> That 2 routers i have problems with are just DNS and DHCP providers,
they
>> are not actibe as default gateway for VMs,
>> there are ~10-12 VMs.
>>
>> router.aggregation.command.each.timeout=3
>>
>>> Hi Daniel,
>>>
>>> At what value is router.aggregation.command.each.timeout global
setting?
>>>
>>> Can you tell a bit about your setup, what the router does (config) as
in
>>> how many VMs, tiers, public IPs etc.
>>>
>>> Regards, Remi
>>>
>>> Sent from my iPhone
>>>
>>>> On 28 Jan 2016, at 05:18, Daniel Mezentsev <dan@soleks.com> wrote:
>>>>
>>>> Hi Remi,
>>>>
>>>> I just upgraded 4.6.2->4.7.1 The same issue, unable to start 2 VRs
(out
>>>> of
>>>> 8 total). VR actually is started, running for 5-7 minutes, then
shuting
>>>> down. One thing that i noticed in the log:
>>>>
>>>> arping -c 1 -I eth0 -A -U -s 10.1.12.1 None
>>>>
>>>> What is "None" here ?
>>>>
>>>> Please let me know what information from the log can i provide you to
>>>> help
>>>> with troubleshooting. I noticed that there are a lot of the same
issues
>>>> happened after upgrade to 4.6, seems like it's getting critical.
>>>>
>>>>> No, just the same template so it's just a matter of upgrading
package.
>>>>> Will ping you when the release is ready.
>>>>>
>>>>> Regards, Remi
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On 25 Jan 2016, at 21:41, Daniel Mezentsev <dan@soleks.com> wrote:
>>>>>>
>>>>>> Hi Remi,
>>>>>>
>>>>>> Absolutely. I don't see any issue to do upgrade to 4.7.1. Will it be
>>>>
>>>> with
>>>>>> new templates for system VMs ?
>>>>>>
>>>>>>> Hi Daniel,
>>>>>>>
>>>>>>> Is upgrading to 4.7.1 (once released tomorrow) an option? It has
>>>>>>> many
>>>>>>> improvements over 4.6.x and I'm quite sure it fixes the problem you
>>>>>>> experience now.
>>>>>>>
>>>>>>> Regards, Remi
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>>> On 25 Jan 2016, at 16:29, Daniel Mezentsev <dan@soleks.com> wrote:
>>>>>>>>
>>>>>>>> in my configuration VR is acting as DHCP/DNS server, so it's not
>>>>>>>> passing
>>>>>>>> any traffic, so i can exclude public IP addresses. Definetely
there
>>>>>>>> is
>>>>>>>> one
>>>>>>>> of the VR script is acting badly.
>>>>>>>>
>>>>>>>>> If I disassociated 10 public IPs and keep 10 public IPs (but keep
>>
>> 20
>>>>>> vms
>>>>>>>>> associated with the network), the network (virtual router)
restart
>>>>>>>>> with
>>>>>>>>> clean up successfully (around in 3 min).
>>>>>>>>> I thinks that is a timeout problem (or VR scripts performance)...
>>>>>>>>> but
>>>>>>>>> which timeout parameter?
>>>>>>>>>
>>>>>>>>>> On 24/01/2016 09:43, Milamber wrote:
>>>>>>>>>> Please note, my installation type is Advanced network without
>>>>>>>>>> security
>>>>>>>>>> groups (vlan isolation).
>>>>>>>>>>
>>>>>>>>>>> On 24/01/2016 06:57, Remi Bergsma wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> We have seen this issue sometimes in the VRs from 4.6 and on.
>>>>>>>>>>> There
>>>>>>>>>>> have been several improvements in the code in 4.7.
>>>>>>>>>>>
>>>>>>>>>>> Also, setting router.aggregation.command.each.timeout global
>>>>
>>>> setting
>>>>>>>>>>> to 15 or 20 also may help (restart mgt server after change).
>>>>>>>>>>>
>>>>>>>>>>> @Milamber can you test this setting with 4.7.1RC please? I
>>
>> haven't
>>>>>>>>>>> seen this issue in my test and production systems.
>>>>>>>>>>>
>>>>>>>>>>> Thanks, Remi
>>>>>>>>>>>
>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>
>>>>>>>>>>>> On 24 Jan 2016, at 00:49, Milamber <milamber@apache.org>
wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> I've have a similiar issue without solution (but not with an
>>>>>>>>>>>> upgrade
>>>>>>>>>>>> from 4.5), please see
>>>>>>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9255
>>>>>>>>>>>>
>>>>>>>>>>>> Milamber
>>>>>>>>>>>>
>>>>>>>>>>>>> On 24/01/2016 03:32, Daniel Mezentsev wrote:
>>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've got some issue after i did cloudstack upgrade 4.5 ->
>>
>> 4.6.2.
>>>>>>>>>>>>> Environment has 8 VR, 6 were upgraded and restarted
>>>>>>>>>>>>> successfully, 2
>>>>>>>>>>>>> of them
>>>>>>>>>>>>> stuck. I removed them - the same result, i can't boot them,
>>>>>>>>>>>>> restarted
>>>>>>>>>>>>> network with clean-up - no luck.
>>>>>>>>>>>>> VR actually is starting, i can ssh to it, but it's shuting
>>>>>>>>>>>>> down
>>>>
>>>> in
>>>>>>>>>>>>> approx 5
>>>>>>>>>>>>> minutes. I can extend it runing
>>>>>>>>>>>>> "init 3" command, but cloudstack is not detecting router
>>>>>>>>>>>>> state.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can bing some VMs within the same subnet, so it's not
>>>>>>
>>>>>> VLAN/network
>>>>>>>>>>>>> issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Completely ran out of ideas.
>>>>>>>>>>>>> Should mention network offering is Isolated network without
>>
>> NAT.
>>>>>>>>>>>>> Any help is more then welcome. I can provide any related log,
>>>>>>>>>>>>> but i
>>>>>>>>>>>>> need
>>>>>>>>>>>>> some heads-up what i should look for.
>>>>>>>>>>>>> Daniel Mezentsev, founder
>>>>>>>>>>>>> (+1) 604 313 8592.
>>>>>>>>>>>>> Soleks Data Group.
>>>>>>>>>>>>> Shaping the clouds.
>>>>>>>>
>>>>>>>> Daniel Mezentsev, founder
>>>>>>>> (+1) 604 313 8592.
>>>>>>>> Soleks Data Group.Shaping the clouds.
>>>>>>
>>>>>> Daniel Mezentsev, founder
>>>>>> (+1) 604 313 8592.
>>>>>> Soleks Data Group.Shaping the clouds.
>>>>
>>>> Daniel Mezentsev, founder
>>>> (+1) 604 313 8592.
>>>> Soleks Data Group.Shaping the clouds.
>>
>> Daniel Mezentsev, founder
>> (+1) 604 313 8592.
>> Soleks Data Group.Shaping the clouds.
>
>  
  Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.

--=_DRVECJvfNul2kNyzbPNFnlt--