cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Weller <swel...@ena.com.INVALID>
Subject Re: Latest Qemu KVM EV appears to be broken with ACS
Date Mon, 15 Apr 2019 13:54:40 GMT
+1 for the qemu guest agent approach.


________________________________
From: Wido den Hollander <wido@widodh.nl>
Sent: Saturday, April 13, 2019 2:32 PM
To: dev@cloudstack.apache.org; Rohit Yadav
Subject: Re: Latest Qemu KVM EV appears to be broken with ACS



On 4/12/19 9:33 PM, Rohit Yadav wrote:
> Thanks, I was already exploring a solution using qemu guest agent since morning today.
It just so happened that you also thought of the approach, and I could validate my script
to work with qemu ev 2.12 by the end of my day.
>

That would be great actually. The Qemu Guest Agent is a lot better to
use. We might want to explore that indeed. Not for now, but it is a
better option to talk to VMs imho.

Wido

> A proper fix might require some additional changes in cloud-early-config and therefore
a new systemvmtemplate for 4.13.0.0/4.11.3.0, I'll start a PR on that in the following week(s).
>
> Regards.
>
> Regards,
> Rohit Yadav
>
> ________________________________
> From: Marcus <shadowsor@gmail.com>
> Sent: Saturday, April 13, 2019 12:31:33 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Latest Qemu KVM EV appears to be broken with ACS
>
> Wow, that was fast. Good work.
>
> The script seems to work for me. There was one case where I rebooted the
> router and got the old link local IP somehow. I'm not sure if that was a
> timing issue in seeing the existing /var/cache/cloud/cmdline before the new
> one was written or what, but if it was a timing issue it would seem like we
> should already have that problem with the existing cloud-early-config.
>
> On Fri, Apr 12, 2019 at 12:24 PM Rohit Yadav <rohit.yadav@shapeblue.com>
> wrote:
>
>> Hi Marcus, Simon,
>>
>>
>> I explore two of the short term solutions and I've a working (work in
>> progress) script that replaces the patchviasocket script to use the qemu
>> guest agent (that is installed in 4.11+ sytemvmtemplate). This was part of
>> a scoping exercise for solving the patching problem for qemu 2.12+ (Ubuntu
>> 19.04 has 3.x version).
>>
>>
>> This is what I've so far, however, further testing is needed:
>>
>> https://gist.github.com/rhtyd/ddb42c4c7581c4129ca04fbb829f16cf
>>
>>
>> The logic is completely written in bash as:
>>
>> - Try if we're able to contact the guest agent
>>
>> - Once we're able to connect, confirm that the I/O is not error prone
>>
>> - Then write the payload as file (the ssh public key and cmdline string)
>>
>> - Then fix file permissions
>> - Hope that internally cloud-early-config would detect the cmdline we had
>> saved and patching would work
>>
>>
>> While this may work, for the long term a proper fix is needed that should
>> be a standard patching mechanism across all hypervisors.
>>
>>
>> Regards,
>>
>> Rohit Yadav
>>
>> Software Architect, ShapeBlue
>>
>> https://www.shapeblue.com
>>
>> ________________________________
>> From: Marcus <shadowsor@gmail.com>
>> Sent: Friday, April 12, 2019 11:30:46 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: Latest Qemu KVM EV appears to be broken with ACS
>>
>> Long ago it was a disk. The problem was that these disks had to go
>> somewhere, a place where they could survive migrations, which didn't work
>> well for block based primary storage... at least for the code base at the
>> time. Using virtio socket was seen as a fairly standard way to communicate
>> temporary information to the guest, and didn't require managing the
>> lifecycle of a special disk.
>>
>> I believe the current problem is that the sender needs to remain connected
>> until the receiver has read. Maybe socat does this, but if so we need to
>> ensure that it is available and applied as a new RPM dependency. In my
>> testing, waiting on the sender side didn't 100% fix things, or sometimes
>> took a very long time due to the backoff algorithm on the
>> cloud-early-config receiver. Some tweaks to that made it more robust, but
>> it is still a game of trying to coordinate timing of two services on either
>> end. If it works though, I'm all for it.
>>
>> Just to throw another idea out there... If we want to fix this without
>> involving storage, I might suggest switching to the qemu-guest-agent that
>> now exists, with a socket and listening client already in the system vm.
>> This would be far more robust, I think, than our scripting reading unix
>> sockets without any sort of protocol or buffer control considerations, and
>> would likely be more robust to changes in qemu as the guest agent is the
>> primary target for the feature.
>>
>> We can directly write our /var/cache/cloud/cmdline from the host like so
>> (I'm using virsh but we could perhaps communicate with the guest agent
>> socket directly or via socat):
>>
>> virsh qemu-agent-command 19 '{"execute":"guest-file-open",
>> "arguments":{"path":"/tmp/testfile","mode":"w+"}}'
>> {"return":1001}
>>
>> virsh qemu-agent-command 19 '{"execute":"guest-file-write",
>> "arguments":{"handle":1001,"buf-b64":"Zm9vIHdhcyBoZXJlCg=="}}'
>> {"return":{"count":13,"eof":false}}
>>
>> virsh qemu-agent-command 19 '{"execute":"guest-file-close",
>> "arguments":{"handle":1001}}'
>> {"return":{}}
>>
>> root@r-54850-VM:~# cat /tmp/testfile
>> foo was here
>>
>> We are also able to detect via libvirt that the qemu guest agent is up and
>> ready. You can see it in the XML when you list a VM.
>>
>> We do need to keep other hypervisors in mind. This is just an option for a
>> fix that doesn't involve a larger redesign.
>>
>> On Fri, Apr 12, 2019 at 10:21 AM Rohit Yadav <rohit.yadav@shapeblue.com>
>> wrote:
>>
>>> Hi Simon,
>>>
>>>
>>> I'm exploring a solution for the same, I've found that the python based
>>> patching script fails to wait for the message to be written on the unix
>>> socket before that the socket is closed. I reckon this could be related
>> to
>>> serial port device handling related changes in qemu-ev 2.12, as the same
>>> mechanism used to work in past versions.
>>>
>>>
>>> I'm exploring/testing a solution where I replace the python based
>> patching
>>> script into a bash one. Can you test the following in your envrionment
>>> (ensure socat is installed), just backup and replace the
>> patchviasocket.py
>>> file with this:
>>>
>>> https://gist.github.com/rhtyd/aab23357fef2d8a530c0e83ec8be10c5
>>>
>>>
>>> The short term solution would be one of the ways to ensure patching works
>>> without much change in the scripts or systemvmtemplate. However, longer
>>> term we need to explore and standardize patching mechanism across all
>>> hypervisors, for example by using a small payload via a config drive iso.
>>>
>>>
>>> Regards,
>>>
>>> Rohit Yadav
>>>
>>> Software Architect, ShapeBlue
>>>
>>> https://www.shapeblue.com
>>>
>>> ________________________________
>>> From: Simon Weller <sweller@ena.com.INVALID>
>>> Sent: Friday, April 12, 2019 8:29:04 PM
>>> To: dev; users
>>> Subject: Latest Qemu KVM EV appears to be broken with ACS
>>>
>>> All,
>>>
>>> After troubleshooting a strange issue with a new lab environment
>>> yesterday, it appears that the patchviasocket functionality we rely on
>> for
>>> key and ip injection into our router/SSVM/CPVM images is broken with
>>> qemu-kvm-ev-2.12.0-18.el7 (January 2019 release). This was tested on
>> Centos
>>> 7.6.
>>> No data is injected and this was confirmed using socat on /dev/vport0p1.
>>> qemu-kvm-ev-2.10.0-21.el7_5.7.1 works, so hopefully this will save
>> someone
>>> some pain and suffering trying to figure out why the deployed seems
>> broken.
>>>
>>> We're going to dig in and see if can figure out the patches responsible
>>> for it breaking.
>>>
>>> -Si
>>>
>>>
>>>
>>> rohit.yadav@shapeblue.com
>>> www.shapeblue.com<http://www.shapeblue.com>
>>> Amadeus House, Floral Street, London  WC2E 9DPUK
>>> @shapeblue
>>>
>>>
>>>
>>>
>>
>> rohit.yadav@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> Amadeus House, Floral Street, London  WC2E 9DPUK
>> @shapeblue
>>
>>
>>
>>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> Amadeus House, Floral Street, London  WC2E 9DPUK
> @shapeblue
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message