Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Fri, 16 Dec 2016 11:46:11 +0800
From: Syahrul Sazli Shaharir <sazli@nocser.net>
To: dev@cloudstack.apache.org
Subject: Re: patchviasocket seems to be broken with qemu 2.3(+?)
Organization: ModernOne Data Solutions Sdn. Bhd.
In-Reply-To: <alpine.LRH.2.11.1612161054300.10580@monitor.nocser.net>
References: <2f6cfd1f-189b-ecff-edc2-c307a8dfdb62@host1plus.com>
 <9a2caf48-6d54-fcb0-0a0a-c5de5e5f705f@host1plus.com>
 <alpine.LRH.2.11.1612161054300.10580@monitor.nocser.net>
Message-ID: <1c8a1e1372123511c68af124ae40bc1a@nocser.net>
User-Agent: Roundcube Webmail/1.1.4
archived-at: Fri, 16 Dec 2016 03:46:24 -0000

On 2016-12-16 11:27, Syahrul Sazli Shaharir wrote:
> On Wed, 26 Oct 2016, Linas ?ilinskas wrote:
> 
>> So after some investigation I've found out that qemu 2.3.0 is indeed 
>> broken, at least the way CS uses the qemu chardev/socket.
>> 
>> Not sure in which specific version it happened, but it was fixed in 
>> 2.4.0-rc3, specifically noting that CloudStack 4.2 was not working.
>> 
>> qemu git commit: 4bf1cb03fbc43b0055af60d4ff093d6894aa4338
>> 
>> Also attaching the patch from that commit.
>> 
>> 
>> For our own purposes i've included the patch to the qemu-kvm-ev 
>> package (2.3.0) and all is well.
> 
> Hi,
> 
> I am facing the exact same issue on latest Cloudstack 4.9.0.1, on
> latest CentOS 7.3.1611, with latest qemu-kvm-ev-2.6.0-27.1.el7
> package.
> 
> The issue initially surfaced following a heartbeat-induced reset of
> all hosts, when it was on CS 4.8 @ CentOS 7.0 and stock
> qemu-kvm-1.5.3. Since then, the patchviasocket.pl/py timeouts
> persisted for 1 out of 4 router VM/networks, even after upgrading to
> latest code. (I have checked the qemu-kvm-ev-2.6.0-27.1.el7 source,
> and the patched code are pretty much still intact, as per the
> 2.4.0-rc3 commit).
> 
> Any help would be greatly appreciated.
> 
> Thanks.
> 
> (Attached are some debug logs from the host's agent.log)

Here are the debug logs as mentioned: http://pastebin.com/yHdsMNzZ

Thanks.

> 
> --sazli
> 
>> 
>> 
>> On 2016-10-20 09:59, Linas ?ilinskas wrote:
>>> 
>>>  Hi.
>>> 
>>>  We have made an upgrade to 4.9.
>>> 
>>>  Custom build packages with our own patches, which in my mind (i'm 
>>> the only
>>>  one patching those) should not affect the issue i'll describe.
>>> 
>>>  I'm not sure whether we didn't notice it before, or it's actually 
>>> related
>>>  to something in 4.9
>>> 
>>>  Basically our system vm's were unable to be patched via the qemu 
>>> socket.
>>>  The script simply error'ed out with a timeout while trying to push 
>>> the
>>>  data to the socket.
>>> 
>>>  Executing it manually (with cmd line from the logs) resulted the 
>>> same. I
>>>  even tried the old perl variant, which also had same result.
>>> 
>>>  So finally we found out that this issue happens only on our HVs 
>>> which run
>>>  qemu 2.3.0, from the centos 7 special interest virtualization repo. 
>>> Other
>>>  ones that run qemu 1.5, from official repos, can patch the system 
>>> vms
>>>  fine.
>>> 
>>>  So i'm wondering if anyone tested 4.9 with kvm with qemu >= 2.x? 
>>> Maybe it
>>>  something else special in our setup. e.g. we're running the HVs from 
>>> a
>>>  preconfigured netboot image (pxe), but all of them, including those 
>>> with
>>>  qemu 1.5, so i have no idea.
>>> 
>>> 
>>>  Linas ?ilinskas
>>>  Head of Development
>>>  website <http://www.host1plus.com/> facebook
>>>  <https://www.facebook.com/Host1Plus> twitter
>>>  <https://twitter.com/Host1Plus> linkedin
>>>  <https://www.linkedin.com/company/digital-energy-technologies-ltd.>
>>> 
>>>  Host1Plus is a division of Digital Energy Technologies Ltd.
>>> 
>>>  26 York Street, London W1U 6PZ, United Kingdom
>>> 
>> 
>> Linas ?ilinskas
>> Head of Development
>> website <http://www.host1plus.com/> facebook 
>> <https://www.facebook.com/Host1Plus> twitter 
>> <https://twitter.com/Host1Plus> linkedin 
>> <https://www.linkedin.com/company/digital-energy-technologies-ltd.>
>> 
>> Host1Plus is a division of Digital Energy Technologies Ltd.
>> 
>> 26 York Street, London W1U 6PZ, United Kingdom
>> 
>> 
>> 

-- 
--sazli


[ HP | Dell | Microsoft | Symantec | Server & Network Infrastructure ]
W : www.modern.com.my