cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohit Yadav <rohit.ya...@shapeblue.com>
Subject Re: Latest Qemu KVM EV appears to be broken with ACS #3278
Date Thu, 25 Jul 2019 11:38:08 GMT
Hi Sven,


ACS 4.12 does not have the fix to make it work with qemu-ev, we only recently got it in 4.11
and it is only available curently with 4.11.3.0 or latest master.


Regards,

Rohit Yadav

Software Architect, ShapeBlue

https://www.shapeblue.com

________________________________
From: Boris Stoyanov <boris.stoyanov@shapeblue.com>
Sent: Thursday, July 25, 2019 3:38:29 PM
To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>
Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278

On my KVM host I have:
CentOS Linux release 7.6.1810 (Core)
Running the following qemu related packages

qemu-img-ev-2.12.0-18.el7_6.5.1.x86_64
libvirt-daemon-driver-qemu-4.5.0-10.el7_6.12.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
centos-release-qemu-ev-1.0-4.el7.centos.noarch
qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64

I was able to run system vms and they connect back to management, also was able to spin user
vms and router

I'm running ACS master (as of one of the latest meged PRs #3504)

Am I missing something?

Bobby.

On 25.07.19, 12:38, "Sven Vogel" <S.Vogel@ewerk.com> wrote:

    Hey Bobby,

    thanks Bobby.

    Which version of QEMU do you use? You mean 4.12 and later? Which CentOS and libvirt?

    We used the actually master, SystemVM and qemu 3.10 and 3.12. it seems it related to the
cloudstack version 4.12 and above…

    Example: qemu-kvm-ev-2.10.0-21.el7_5.7.1 / CentOS 7 / actually master / SystemVM before
4.11.X

    --VR Dumpxml (two channel, first in use from libvirt, I know its disconnected but it seems
here like the same not online)
        <channel type='unix'>
          <source mode='bind' path='/var/lib/libvirt/qemu/r-1741-VM.agent'/>
          <target type='virtio' name='r-1712-VM.vport' state='disconnected'/>
          <alias name='channel0'/>
          <address type='virtio-serial' controller='0' bus='0' port='1'/>
        </channel>
        <channel type='unix'>
          <source mode='bind' path='/var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0'/>
          <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
          <alias name='channel1'/>
          <address type='virtio-serial' controller='0' bus='0' port='2'/>
        </channel>

    — Socket seems there...
    [root@kvm08 /]# ll /var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0
    srwxrwxr-x. 1 root root 0 Jul 23 11:49 /var/lib/libvirt/qemu/r-1741-VM.org.qemu.guest_agent.0

    —related to the channel it seems to work
    [root@kvm08 /]# virsh qemu-agent-command r-1741-VM '{"execute":"guest-ping"}'
    {"return":{}}

    Example: qemu-kvm-ev-2.10.0-21.el7_5.7.1 / CentOS 7 / actually master / SystemVM 4.12.0
or later

    —VR Dumpxml
        <channel type='unix'>
          <source mode='bind' path='/var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0'/>
          <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
          <alias name='channel0'/>
          <address type='virtio-serial' controller='0' bus='0' port='1'/>
        </channel>

    — Socket seems there…
    [root@kvm08 /]# ll /var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0
    srwxrwxr-x. 1 root root 0 Jul 24 22:23 /var/lib/libvirt/qemu/r-1750-VM.org.qemu.guest_agent.0

    —related to the channel it seems not to work
    [root@kvm08 /]# virsh qemu-agent-command r-1750-VM '{"execute":"guest-ping"}'
    error: Guest agent is not responding: QEMU guest agent is not connected

    After „second" restart of the systemVM the agent gets connected, patch.sh is running
and cloud-early-config does configure the machine.
    The second channel is always connected from the first boot.
    The curious thing is that if we reboot the vpc provisioning is done correctly and the
first channel is up too.

    It seems that libvirt use the first channel for its own. before it was a vport and agent.0
… but it seems only the second channel was used.

    —>https://wiki.libvirt.org/page/Qemu_guest_agent
    Configure guest agent without libvirt interference

    "In some cases, users might want to configure a guest agent in their domain XML but don't
want libvirt to connect to guest agent socket at all. Because libvirt connects to a guest
agent channel if and only if it is a virtio channel with org.qemu.guest_agent.0 name, all
one need to do is void at least one of these conditions. However, the most feasible way is
to change the target's name:"

    It seems we need to configure a second one as org.qemu.guest_agent.1 or use two of the
channels!

    1x org.qemu.guest_agent.1
    or
    org.qemu.guest_agent.0
    And
    org.qemu.guest_agent.1

    Maybe I am wrong. Its only a theory that needs to be more investigation …

    I think we have a problem with the channels or qemu agent inside the SystemVM.

    Cheers,

    Sven



    __

    Sven Vogel
    Teamlead Platform

    EWERK DIGITAL GmbH
    Brühl 24, D-04109 Leipzig
    P +49 341 42649 - 11
    F +49 341 42649 - 18
    S.Vogel@ewerk.com
    www.ewerk.com<http://www.ewerk.com>

    Geschäftsführer:
    Dr. Erik Wende, Hendrik Schubert, Frank Richter
    Registergericht: Leipzig HRB 17023

    Zertifiziert nach:
    ISO/IEC 27001:2013
    DIN EN ISO 9001:2015
    DIN ISO/IEC 20000-1:2011

    ISAE 3402 Typ II Assessed

    EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group>
| Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group>
| Facebook<https://de-de.facebook.com/EWERK.IT/>

    Mit Handelsregistereintragung vom 09.07.2019 ist die EWERK RZ GmbH auf die EWERK IT GmbH
verschmolzen und firmiert nun gemeinsam unter dem Namen: EWERK DIGITAL GmbH, für weitere
Informationen klicken Sie hier<https://www.ewerk.com/ewerkdigital>.

    Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

    Disclaimer Privacy:
    Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich
und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger
sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie
die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

    The contents of this e-mail (including any attachments) are confidential and may be legally
privileged. If you are not the intended recipient of this e-mail, any disclosure, copying,
distribution or use of its contents is strictly prohibited, and you should please notify the
sender immediately and then delete it (including any attachments) from your system. Thank
you.

    Am 24.07.2019 um 09:47 schrieb Boris Stoyanov <boris.stoyanov@shapeblue.com<mailto:boris.stoyanov@shapeblue.com>>:

    Hi Sven,

    I've just installed master and upgraded my hosts to latest qemu-ev, then was able to deploy
successfully the system vms, so looks like it's related to 4.12.

    Bobby.

    On 21.07.19, 18:23, "Sven Vogel" <S.Vogel@ewerk.com<mailto:S.Vogel@ewerk.com>>
wrote:

       Hi Rohit,

       Thx for input. We don’t  checked the 11.X.X version. We tied the upcoming master
and 4.12. we check in the following week the XML from a actual Version (Master). Maybe you
are right and we see only one. I think it’s a good idea to rid of the old adapter. I think
what you mean is we should see only „path='/var/lib/libvirt/qemu/r-1710-VM.org.qemu.guest_agent.0'/><target
type='virtio' name='org.qemu.guest_agent.0'/>“ that one. But why we don’t get an connection
to the router/systemVM. So what i see is that the Agent socket it present. We will try the
„virsh qemu-agent-command“ and see how if we get an connection.

       Do you think the problem is a Agent Problem into the SystemVM?

       Cheers

       Sven



       __

       Sven Vogel
       Teamlead Platform

       EWERK RZ GmbH
       Brühl 24, D-04109 Leipzig
       P +49 341 42649 - 11
       F +49 341 42649 - 18
       S.Vogel@ewerk.com<mailto:S.Vogel@ewerk.com>
       www.ewerk.com<http://www.ewerk.com/>

       Geschäftsführer:
       Dr. Erik Wende, Hendrik Schubert, Frank Richter
       Registergericht: Leipzig HRB 17023

       Zertifiziert nach:
       ISO/IEC 27001:2013
       DIN EN ISO 9001:2015
       DIN ISO/IEC 20000-1:2011

       EWERK-Blog | LinkedIn | Xing | Twitter | Facebook

       Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

       Disclaimer Privacy:
       Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich
und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger
sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie
die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

       The contents of this e-mail (including any attachments) are confidential and may be
legally privileged. If you are not the intended recipient of this e-mail, any disclosure,
copying, distribution or use of its contents is strictly prohibited, and you should please
notify the sender immediately and then delete it (including any attachments) from your system.
Thank you.
    Am 21.07.2019 um 16:50 schrieb Rohit Yadav <rohit.yadav@shapeblue.com<mailto:rohit.yadav@shapeblue.com>>:

    Hi Sven,


    Are you able to reproduce the failure against latest KVM qemu-ev + 4.11.3 systemvmtemplate
with ACS 4.11.3.0? The PR https://github.com/apache/cloudstack/pull/3278 only fixes the issue
towards 4.11.3.0 and upcoming master/4.13.0.0, qemu-ev will fail for 4.11.2.0 or 4.12.0.0.
Also, with the PR/patch applied you should only see a single virtio serial port device on
the domain xml, we got rid of the old/special case which would add a specific vport/serial-port
device for systemvms.


    Regards,

    Rohit Yadav

    Software Architect, ShapeBlue

    https://www.shapeblue.com<https://www.shapeblue.com/>

    ________________________________
    From: Sven Vogel <S.Vogel@ewerk.com<mailto:S.Vogel@ewerk.com>>
    Sent: Saturday, July 20, 2019 3:45:08 PM
    To: dev <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
    Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278

    Hi Simon,

    Thanks for your information. We looking forward it you on a more recent mainline version.
In the meantime we try find out more out more about this bug.

    Please feel free if you have new information add them here.

    In the meantime I wrote some comment below this PR #3278

    Cheers

    Sven


    __

    Sven Vogel
    Teamlead Platform

    EWERK DIGITAL GmbH
    Brühl 24, D-04109 Leipzig
    P +49 341 42649 - 11
    F +49 341 42649 - 18
    S.Vogel@ewerk.com<mailto:S.Vogel@ewerk.com>
    www.ewerk.com<http://www.ewerk.com>

    Geschäftsführer:
    Dr. Erik Wende, Hendrik Schubert, Frank Richter
    Registergericht: Leipzig HRB 17023

    Zertifiziert nach:
    ISO/IEC 27001:2013
    DIN EN ISO 9001:2015
    DIN ISO/IEC 20000-1:2011

    ISAE 3402 Typ II Assessed

    EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group>
| Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group>
| Facebook<https://de-de.facebook.com/EWERK.IT/>

    Mit Handelsregistereintragung vom 10.07.2019 ist die EWERK RZ GmbH auf die EWERK IT GmbH
verschmolzen und firmiert zukünftig gemeinsam unter dem Namen: EWERK DIGITAL GmbH, für weitere
Informationen klicken Sie hier<https://www.ewerk.com/ewerkdigital>.

    Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

    Disclaimer Privacy:
    Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich
und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger
sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie
die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

    The contents of this e-mail (including any attachments) are confidential and may be legally
privileged. If you are not the intended recipient of this e-mail, any disclosure, copying,
distribution or use of its contents is strictly prohibited, and you should please notify the
sender immediately and then delete it (including any attachments) from your system. Thank
you.

    Am 19.07.2019 um 15:04 schrieb Simon Weller <sweller@ena.com.INVALID<mailto:sweller@ena.com.INVALID>>:

    Hi Sven,

    We're still using 2.10 right now and we haven't tested the patches yet for 2.12. Having
said that, we're currently trying to get our internal ACS release up to a more recent mainline
version, so I'd suspect we'll have to dive into this fairly soon.

    -Si

    ________________________________
    From: Sven Vogel <S.Vogel@ewerk.com<mailto:S.Vogel@ewerk.com>>
    Sent: Friday, July 19, 2019 6:48 AM
    To: dev <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
    Subject: Re: Latest Qemu KVM EV appears to be broken with ACS #3278

    Hi Guys,

    Sorry that I formal reopen this issue.

    We tested the actual system with this patch #3278

    We use CentOS 1805 or 1810 and OVS and QEMU 2.12. It seems the first agent contact will
not work so that the agent know the virtual machine is running. With 2.10 and the old agent,
SystemVM it works.

    2019-07-19 00:17:30,571 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null)
(logid:8ad14091) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh
-n v-1728-VM -c  template=domP type=consoleproxy host=10.24.48.46 port=8250 name=v-1728-VM
premium=true zone=4 pod=4 guid=Proxy.1728 proxy_vm=1728 disable_rp_filter=true eth2ip=185.232.219.237
eth2mask=255.255.255.248 gateway=185.232.219.233 eth0ip=169.254.0.193 eth0mask=255.255.0.0
eth1ip=10.24.48.124 eth1mask=255.255.255.0 mgmtcidr=10.24.48.0/24 localgw=10.24.48.1 internaldns1=10.24.48.33
internaldns2=10.24.48.34 dns1=217.69.224.73 dns2=85.232.28.146
    2019-07-19 00:17:30,573 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null)
(logid:8ad14091) Executing while with timeout : 300000
    2019-07-19 00:17:30,616 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null)
(logid:7577151e) Unable to logon to 169.254.2.189
    2019-07-19 00:17:30,616 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null)
(logid:7577151e) Trying to connect to 169.254.2.189
    2019-07-19 00:17:31,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null)
(logid:04b4d6ce) Could not connect to 169.254.1.21
    2019-07-19 00:17:33,622 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null)
(logid:7577151e) Could not connect to 169.254.2.189
    2019-07-19 00:17:36,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null)
(logid:04b4d6ce) Unable to logon to 169.254.1.21
    2019-07-19 00:17:36,874 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null)
(logid:04b4d6ce) Trying to connect to 169.254.1.21
    2019-07-19 00:17:38,622 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-2:null)
(logid:7577151e) Trying to connect to 169.254.2.189

    @simonweller you encountered the same problem?

    @rohit maybe there is something what we forget or what’s wrong?

    @wido do you have ideas?

    thanks

    Sven


    __

    Sven Vogel
    Teamlead Platform

    EWERK DIGITAL GmbH
    Brühl 24, D-04109 Leipzig
    P +49 341 42649 - 11
    F +49 341 42649 - 18
    S.Vogel@ewerk.com<mailto:S.Vogel@ewerk.com>
    www.ewerk.com<http://www.ewerk.com/><http://www.ewerk.com<http://www.ewerk.com/>>

    Geschäftsführer:
    Dr. Erik Wende, Hendrik Schubert, Frank Richter
    Registergericht: Leipzig HRB 17023

    Zertifiziert nach:
    ISO/IEC 27001:2013
    DIN EN ISO 9001:2015
    DIN ISO/IEC 20000-1:2011

    ISAE 3402 Typ II Assessed

    EWERK-Blog<https://blog.ewerk.com/> | LinkedIn<https://www.linkedin.com/company/ewerk-group>
| Xing<https://www.xing.com/company/ewerk> | Twitter<https://twitter.com/EWERK_Group>
| Facebook<https://de-de.facebook.com/EWERK.IT/>

    Mit Handelsregistereintragung vom 10.07.2019 ist die EWERK RZ GmbH auf die EWERK IT GmbH
verschmolzen und firmiert zukünftig gemeinsam unter dem Namen: EWERK DIGITAL GmbH, für weitere
Informationen klicken Sie hier<https://www.ewerk.com/ewerkdigital>.

    Auskünfte und Angebote per Mail sind freibleibend und unverbindlich.

    Disclaimer Privacy:
    Der Inhalt dieser E-Mail (einschließlich etwaiger beigefügter Dateien) ist vertraulich
und nur für den Empfänger bestimmt. Sollten Sie nicht der bestimmungsgemäße Empfänger
sein, ist Ihnen jegliche Offenlegung, Vervielfältigung, Weitergabe oder Nutzung des Inhalts
untersagt. Bitte informieren Sie in diesem Fall unverzüglich den Absender und löschen Sie
die E-Mail (einschließlich etwaiger beigefügter Dateien) von Ihrem System. Vielen Dank.

    The contents of this e-mail (including any attachments) are confidential and may be legally
privileged. If you are not the intended recipient of this e-mail, any disclosure, copying,
distribution or use of its contents is strictly prohibited, and you should please notify the
sender immediately and then delete it (including any attachments) from your system. Thank
you.

    Am 22.04.2019 um 21:29 schrieb Simon Weller <sweller@ena.com.INVALID<mailto:sweller@ena.com.INVALID><mailto:sweller@ena.com.INVALID>>:

    In our case the SystemVMs were booting fine, but ACS wasn't able to inject the payload
via the socket.


    rohit.yadav@shapeblue.com
    www.shapeblue.com<http://www.shapeblue.com>
    Amadeus House, Floral Street, London  WC2E 9DPUK
    @shapeblue






    boris.stoyanov@shapeblue.com<mailto:boris.stoyanov@shapeblue.com>
    www.shapeblue.com<http://www.shapeblue.com/>
    Amadeus House, Floral Street, London  WC2E 9DPUK
    @shapeblue







boris.stoyanov@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue




rohit.yadav@shapeblue.com 
www.shapeblue.com
Amadeus House, Floral Street, London  WC2E 9DPUK
@shapeblue
  
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message