cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: VPC's VR missing public NIC eth1
Date Fri, 30 May 2014 12:49:40 GMT
Hi Joris,

I have turned on DEBUG loging in agent.log on cs1.xxx/net host:

So, management logs again: http://pastebin.com/F6BRf7Y9
Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC

Not playing smart, but there is some error: [kvm.resource.KVMGuestOsMapper]
(agentRequest-Handler-3:) Can't find the mapping of guest os: Debian
GNU/Linux 7(64-bit)

Best,
Andrija


On 30 May 2014 14:26, Joris van Lieshout <JvanLieshout@schubergphilis.com>
wrote:

> Hi Andrija,
>
> Bold formatting does not come trough on the dev list. :)
> But u might need a bit more info.
>
> At a certain point I see this line
>
> 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request]
> (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 1-609104082: Sending  {
> Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM"...
>
>
> This is where the information is passed on to the agent handles. For XS
> this would initiate an agent handler on the management server but for KVM,
> if I remember correctly, it passed the command on to the cloudstack agent
> service on the hypervisor.
>
> Can you check the cloud service log on the KVM hypervisor executing the
> request? it's this server cs1.xxxxx.net and then search top down for
> 609104082 in the log. See if you can provide the log from the agent
> handler thread started by that sequence.
>
> Kind regards,
> Joris van Lieshout
>
> Schuberg Philis
> Boeingavenue 271
> 1119 PD  Schiphol-Rijk
> schubergphilis.com
>
> +31 20-7506672
> +31 6-51428188
>
>
>
>
> On 30/05/14 14:08, "Andrija Panic" <andrija.panic@gmail.com> wrote:
>
> >Hi Joris,
> >
> >here is the management log: http://pastebin.com/zxnKxFhk
> >
> >Interesting parts (to me): in bold
> >
> >2014-05-30 13:56:21,899 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) copyAsync inspecting src type
> >TEMPLATE copyAsync inspecting dest type VOLUME
> >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Sending  {
> >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net), Ver: v1, Flags:
> 100011,
> >[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apach
> >e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0f-b0b4
> >-5e351e7cae55","origUrl":"
> >
> http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mast
> >er-kvm.qcow2.bz2
> >","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"QCOW2",
> >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","
> >*hvm":true*
> >,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.cloudstack
> >.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-2203f79b1
> >dc6","id":209,"poolType":"RBD","host":"
> >cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD://
> >
> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88
> >a8-2203f79b1dc6
> >"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorType":"
> >KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid"
> >:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataStore":{"
> >org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a6
> >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
> >cephmon.xxxxx.net","path":"cloudstack","port":6789,"url":"RBD://
> >
> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-88
> >a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":1064,"vm
> >Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":0,"hyp
> >ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> >}
> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
> >(AgentManager-Handler-12:null) Seq 4-1248669612: Processing:  { Ans: ,
> >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10,
> >[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.a
> >pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"9c4
> >40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id":0}},"
> >result":true,"wait":0}}]
> >}
> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq 4-1248669612: Received:  {
> >Ans: , MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10, { CopyCmdAnswer
> >} }
> >2014-05-30 13:56:23,773 DEBUG
> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic NicProfile[
> >*1092-801-null*-46.232.xxx.246-vlan://untagged of type Public from the
> >nics
> >passed on vm start. The nic will be plugged later
> >2014-05-30 13:56:23,773 DEBUG
> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
> >NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vlan://4
> >4
> >of type Guest from the nics passed on vm start. The nic will be plugged
> >later
> >2014-05-30 13:56:23,773 DEBUG
> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
> >NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vlan://4
> >3
> >of type Guest from the nics passed on vm start. The nic will be plugged
> >later
> >2014-05-30 13:56:23,773 DEBUG
> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
> >NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3
> >004
> >of type Guest from the nics passed on vm start. The nic will be plugged
> >later
> >2014-05-30 13:56:23,773 DEBUG
> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Removing nic
> >NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vlan://3
> >004
> >of type Guest from the nics passed on vm start
> >
> >
> >Thanks,
> >
> >
> >On 30 May 2014 13:54, Joris van Lieshout <JvanLieshout@schubergphilis.com
> >
> >wrote:
> >
> >> Hi Andrija,
> >>
> >> Just the start of the VR should be sufficient.
> >>
> >> Kind regards,
> >> Joris van Lieshout
> >>
> >> Schuberg Philis
> >> Boeingavenue 271
> >> 1119 PD  Schiphol-Rijk
> >> schubergphilis.com
> >>
> >> +31 20-7506672
> >> +31 6-51428188
> >>
> >>
> >>
> >>
> >> On 30/05/14 13:48, "Andrija Panic" <andrija.panic@gmail.com> wrote:
> >>
> >> >Hi Joris,
> >> >
> >> >just to be sure - you want me to capture the log from the moment I
> >>reboot
> >> >router - or you want me to stop it, then start capturing log, and
> >>start it
> >> >(and continue capture untill ethnull errors inside VR) ?
> >> >
> >> >Thanks,
> >> >
> >> >
> >> >On 30 May 2014 13:39, Joris van Lieshout
> >><JvanLieshout@schubergphilis.com
> >> >
> >> >wrote:
> >> >
> >> >> Hi Andrija,
> >> >>
> >> >> Thanks for the answers. In deed your situation is different so
> >>PV/HVM is
> >> >> not the issue.
> >> >>
> >> >> When reading back the log output you have provided I noted that the
> >>VR
> >> >> messages log indicates that it's waiting for ethnull to be up. This
> >> >>raises
> >> >> the question where null was introduced instead of 1. The ACS
> >>management
> >> >> log output you send was, what I think, later down the road where ACS
> >> >>gives
> >> >> up trying to wait for the VR to come up. If you would capture the
> >> >> job-executor in the management log from startCommand till the
> >>exception,
> >> >> do you see anywhere a mention of ethnull? You might need to reed into
> >> >>the
> >> >> DirectAgent executing the startCommand to find a clue. The thing is
> >> >>that I
> >> >> only have experience with XS based environment so I cannot point you
> >>to
> >> >> the exact output to look for. On XS, at least, it is
> >> >> "[c.c.h.x.r.CitrixResourceBase] (DirectAgent-351:ctx-4a51bb9e)
> >>Created a
> >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1"
> >> >>
> >> >> Kind regards,
> >> >> Joris van Lieshout
> >> >>
> >> >> Schuberg Philis
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 30/05/14 10:48, "Andrija Panic" <andrija.panic@gmail.com>
wrote:
> >> >>
> >> >> >Hi Deen,
> >> >> >no, in DB there is field "vlan_id" with value "untagged" - that
> >> >> >"vlan://untagged" is shown from ACS gui, and is used in API call
(or
> >> >> >better
> >> >> >said commands that are seen in management server logs).
> >> >> >
> >> >> >Best,
> >> >> >Andrija
> >> >> >
> >> >> >
> >> >> >On 30 May 2014 10:37, Daan Hoogland <daan.hoogland@gmail.com>
> wrote:
> >> >> >
> >> >> >> Andrija,
> >> >> >>
> >> >> >> Do not just assign a second net vlan://500 You have one like
that
> >>and
> >> >> >> you don't want conflicting nets using the same vlan. I am
> >>wondering
> >> >> >> why 'untagged' comes out as 'vlan://untagged'. I think that
is the
> >> >> >> bug. Did you find the string 'vlan://untagged' in your db?
> >> >> >>
> >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija Panic
> >> >> >><andrija.panic@gmail.com>
> >> >> >> wrote:
> >> >> >> > Hi Joris,
> >> >> >> >
> >> >> >> > thank you for taking time to address this issue :)
> >> >> >> >
> >> >> >> > So...:
> >> >> >> >
> >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by Inktank for
CEPH
> >> >>support),
> >> >> >>OS
> >> >> >> is
> >> >> >> > Centos 6.5, libvirt 1.2.3 compiled.
> >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was fine
> >> >> >> > - not XS, so I guess no answers for this part :)
> >> >> >> > - guest_os_id is 184 = Debian 7 x64
> >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 = Debian 7 x64
> >> >> >> >
> >> >> >> > This worked previously on 4.2.1 = template was ofcourse
> >> >> >>systemvm-kvm-4.2
> >> >> >> -
> >> >> >> > but that was also Debian 7 x64 type... so this should
not be the
> >> >> >>issues
> >> >> >> > (guest not supported by host...)
> >> >> >> >
> >> >> >> > The only thing that might be out of "standard" = all
SVMs are on
> >> >>CEPH
> >> >> >>-
> >> >> >> > there are official docs on altering database to make
some new
> >> >>System
> >> >> >> > Offering as default for SSVM and CPVM - what I did, I
also have
> >> >>done
> >> >> >>same
> >> >> >> > config in DB, to make VR use another System Offering
as default
> >>-
> >> >> >>which
> >> >> >> is
> >> >> >> > NOT explained in the docs - you could use "Change Offering..."
> >> >>button
> >> >> >>on
> >> >> >> > exiting, shutdown VR to change it per docs...
> >> >> >> > But still this worked all fine on 4.2.1...
> >> >> >> >
> >> >> >> > - regarding /var/cache/cloud/cmdline the content is folowing
at
> >>the
> >> >> >> moment
> >> >> >> > root@r-801-VM:~# cat /var/cache/cloud/cmdline
> >> >> >> > vpccidr=10.0.0.0/8 domain=cscloud.internal dns1=8.8.8.8
dns2=
> >> >> >> template=domP
> >> >> >> > name=r-801-VM eth0ip=169.254.0.75 eth0mask=255.255.0.0
> >> >>type=vpcrouter
> >> >> >> > disable_rp_filter=true
> >> >> >> >
> >> >> >> > Also please note that only eth1 does not have IP info,
eth0
> >> >>(control
> >> >> >> > 169.xxx) and all other eh2 and up that are used for Tiers
get IP
> >> >>info
> >> >> >> fine.
> >> >> >> > I could also manually add IP for eth1 (public NIC) and
start
> >>ifup
> >> >> >>eth1 -
> >> >> >> > and it works fine, but adding new IP Port Forwarding
etc does
> >>not
> >> >> >>work...
> >> >> >> >
> >> >> >> > Daan or somebody said it could be realted to my "Public"
network
> >> >>(in
> >> >> >>the
> >> >> >> > Zones, Physical Network, eth1 listing) is NOT tagged
> >> >> >>(vlan://untagged)...
> >> >> >> > Interestingly the only VR that does work fine is the
VR used in
> >> >>Shared
> >> >> >> > network, but that VR is using IP from Guest IP range
(also
> >> >>efectively
> >> >> >> > public IPs but on vlan 500)
> >> >> >> >
> >> >> >> > I was instructed to try to change Public IP range from
untagged
> >>to
> >> >> >>vlan
> >> >> >> > 500, but I'm not sure how to do this, if there is any
way at all
> >> >> >>(editing
> >> >> >> > "vlan" table and changing to vlan 500 does not work,
after
> >> >>rebooting
> >> >> >>VR
> >> >> >> > from ACS gui).
> >> >> >> >
> >> >> >> > :)
> >> >> >> >
> >> >> >> > So, not sure what is roughly expected date for 4.4, but
right
> >>now,
> >> >>I'm
> >> >> >> > pretty stuck with a big problem of all VPC not operational
at
> >> >>all...
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 30 May 2014 08:27, Joris van Lieshout <
> >> >> >> JvanLieshout@schubergphilis.com>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> >> Hi Andrija,
> >> >> >> >>
> >> >> >> >> Daan asked me to have a look at this as well. Looking
at you
> >> >>issue I
> >> >> >> >> recall having seen something similar. Back then when
upgrading
> >> >>4.2.1
> >> >> >>to
> >> >> >> >> 4.3 I though it had to do with out own custom build
svm
> >>template.
> >> >> >> >> Let me fire off some questions before explaining
what the cause
> >> >>was
> >> >> >>in
> >> >> >> our
> >> >> >> >> case. :)
> >> >> >> >>
> >> >> >> >> - what hypervisor (and version) are you using?
> >> >> >> >> - if XS, is the new VR a para-virtualised instance
(PV) or
> >> >>hardware
> >> >> >> >> assisted (HVM)? Do a "xe vm-param-list" on the VR
uuid and
> >>check
> >> >>that
> >> >> >> >> param PV-args is set and HVM-boot-policy is unset.
> >> >> >> >> - what is the OS type of the VR in ACS (guest_os_id
in
> >>vm_instance
> >> >> >>table
> >> >> >> >> and match with table guest_os)
> >> >> >> >> - what is the OS type of the SVM template?
> >> >> >> >>
> >> >> >> >> Now for the explaining. :)
> >> >> >> >> In our case the OS type of the new template was not
supported
> >>on
> >> >>the
> >> >> >> >> XenServer version we are running. Therefore the VR
was started
> >>by
> >> >>XS
> >> >> >>as
> >> >> >> a
> >> >> >> >> HVM guest. System vms on XS rely on the arguments
passed to
> >>them
> >> >>in
> >> >> >>the
> >> >> >> >> PV-args param (ends up on the guest in /var/cache/cloud/cmdline
> >> >> >>which in
> >> >> >> >> turn is used by cloud-early-config) in order to work.
cmdline
> >> >> >>contains
> >> >> >> the
> >> >> >> >> NIC configuration information.
> >> >> >> >> So, long story short, if a VR gets started as a HVM
it will not
> >> >>get
> >> >> >>the
> >> >> >> >> information needed to configure it's NICs.
> >> >> >> >>
> >> >> >> >> Workaround
> >> >> >> >> We corrected the os_type_id in the DB (yes I know
editing the
> >>DB
> >> >>is
> >> >> >> >> something you usually don't want but there is no
other way in
> >>this
> >> >> >>case)
> >> >> >> >> of the existing VR's and of the systemvmtemplate
to something
> >> >> >>supported
> >> >> >> by
> >> >> >> >> XenServer.
> >> >> >> >>
> >> >> >> >> Kind regards,
> >> >> >> >> Joris van Lieshout
> >> >> >> >>
> >> >> >> >> Schuberg Philis
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On 29/05/14 12:18, "Andrija Panic" <andrija.panic@gmail.com>
> >> >>wrote:
> >> >> >> >>
> >> >> >> >> >They are 2 traffic types on 1 physical net (that
is both
> >>tagged
> >> >>vlan
> >> >> >> 500,
> >> >> >> >> >and untagged packets travel over same KVM bridge,
and over
> >>eth1
> >> >>to
> >> >> >> outside
> >> >> >> >> >world)...
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland <daan.hoogland@gmail.com>
> >> >> wrote:
> >> >> >> >> >
> >> >> >> >> >> Are these two traffic types in one physical
net? or two
> >> >>physical
> >> >> >>nets
> >> >> >> >> >> on the same interface (seems wrong).
> >> >> >> >> >>
> >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM, Jayapal
Reddy Uradi
> >> >> >> >> >> <jayapalreddy.uradi@citrix.com> wrote:
> >> >> >> >> >> > I don't think editing DB table will
work.
> >> >> >> >> >> >
> >> >> >> >> >> > -Jayapal
> >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija
Panic
> >> >> >><andrija.panic@gmail.com
> >> >> >> >
> >> >> >> >> >> wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >> It's like this:
> >> >> >> >> >> >>
> >> >> >> >> >> >> I have public subnet /24.
> >> >> >> >> >> >>
> >> >> >> >> >> >> half is dedicated for Guest traffic
(vlan 500) and the
> >> >>second
> >> >> >> half is
> >> >> >> >> >> >> dedicated to Public traffic/network
(no vlan tags, that
> >>is
> >> >> >> untagged
> >> >> >> >> >> packets)
> >> >> >> >> >> >>
> >> >> >> >> >> >> Both vlan500 and untagged packets
travel over physical
> >>eth1
> >> >> >> >> >>interface on
> >> >> >> >> >> >> hypervisors and can reach Internet.
> >> >> >> >> >> >>
> >> >> >> >> >> >> Thanks,
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> On 29 May 2014 11:06, Daan Hoogland
> >> >><daan.hoogland@gmail.com>
> >> >> >> wrote:
> >> >> >> >> >> >>
> >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57
AM, Andrija Panic <
> >> >> >> >> >> andrija.panic@gmail.com>
> >> >> >> >> >> >>> wrote:
> >> >> >> >> >> >>>> 500
> >> >> >> >> >> >>>
> >> >> >> >> >> >>>
> >> >> >> >> >> >>> is 500 the vlan of your guestnetwork
or your physical
> >> >>network?
> >> >> >> You
> >> >> >> >> >> >>> wouldn't want to have two nets
with vlan 500!
> >> >> >> >> >> >>>
> >> >> >> >> >> >>> --
> >> >> >> >> >> >>> Daan
> >> >> >> >> >> >>>
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> --
> >> >> >> >> >> >>
> >> >> >> >> >> >> Andrija Panić
> >> >> >> >> >> >> --------------------------------------
> >> >> >> >> >> >>  http://admintweets.com
> >> >> >> >> >> >> --------------------------------------
> >> >> >> >> >> >
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >> Daan
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >--
> >> >> >> >> >
> >> >> >> >> >Andrija Panić
> >> >> >> >> >--------------------------------------
> >> >> >> >> >  http://admintweets.com
> >> >> >> >> >--------------------------------------
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> >
> >> >> >> > Andrija Panić
> >> >> >> > --------------------------------------
> >> >> >> >   http://admintweets.com
> >> >> >> > --------------------------------------
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Daan
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> >--
> >> >> >
> >> >> >Andrija Panić
> >> >> >--------------------------------------
> >> >> >  http://admintweets.com
> >> >> >--------------------------------------
> >> >>
> >> >>
> >> >
> >> >
> >> >--
> >> >
> >> >Andrija Panić
> >> >--------------------------------------
> >> >  http://admintweets.com
> >> >--------------------------------------
> >>
> >>
> >
> >
> >--
> >
> >Andrija Panić
> >--------------------------------------
> >  http://admintweets.com
> >--------------------------------------
>
>


-- 

Andrija Panić
--------------------------------------
  http://admintweets.com
--------------------------------------

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message