cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@gmail.com>
Subject Re: VPC's VR missing public NIC eth1
Date Fri, 30 May 2014 16:26:39 GMT
sorry for the flood of mails, I am thinking while typing and clicking.
I will let it sink in over the weekend.

On Fri, May 30, 2014 at 6:25 PM, Daan Hoogland <daan.hoogland@gmail.com> wrote:
> and
> broadcastUriToNicNum.put(
> BroadcastDomainType.Vlan.toUri(Vlan.UNTAGGED).toString(),
> devNum);
>
> ahould read
>
> broadcastUriToNicNum.put(Vlan.UNTAGGED,devNum);
>
>
>
> On Fri, May 30, 2014 at 6:24 PM, Daan Hoogland <daan.hoogland@gmail.com> wrote:
>> Or maybe it shouldn't be filtered by BroadcastDomainType.VLAN when it
>> is 'untagged', ...
>>
>> On Fri, May 30, 2014 at 6:23 PM, Daan Hoogland <daan.hoogland@gmail.com> wrote:
>>> BroadcastDomainType.Vlan.
>>> toUri(Vlan.UNTAGGED).toString() should return 'untagged' in my opinion
>>>
>>> On Fri, May 30, 2014 at 6:21 PM, Marcus <shadowsor@gmail.com> wrote:
>>>> The issue is that we put this in the map:
>>>>
>>>> broadcastUriToNicNum.put(BroadcastDomainType.Vlan.toUri(Vlan.UNTAGGED).toString(),
>>>> devNum);
>>>>
>>>> Which is 'vlan://untagged', and we fetch the eth devNum back by doing:
>>>>
>>>> broadcastUriToNicNum.get(ip.getBroadcastUri());
>>>>
>>>> which is just 'untagged'
>>>>
>>>>
>>>> On Fri, May 30, 2014 at 10:19 AM, Marcus <shadowsor@gmail.com> wrote:
>>>>
>>>>> It's not valid if you've got code that says does string 'vlan://untagged'
>>>>> equal 'untagged'.
>>>>>
>>>>>
>>>>> On Fri, May 30, 2014 at 10:17 AM, Daan Hoogland <daan.hoogland@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> CLOUDSTACK-5505 looks alright. As for the solution; Isn't 'untagged' a
>>>>>> valid uri in itself? I would expect it would have always the value
>>>>>> without the 'vlan://'. That said a solution is better then no
>>>>>> solution, maybe the db upgrade path is best.
>>>>>>
>>>>>> Andrija, can you try as Marcus suggests, editing the db to change
>>>>>> 'untagged' to 'vlan://untagged'?
>>>>>>
>>>>>> thanks,
>>>>>> Daan
>>>>>>
>>>>>> On Fri, May 30, 2014 at 6:03 PM, Marcus <shadowsor@gmail.com> wrote:
>>>>>> > If that works, then I think the fix is better to update the database
>>>>>> > upgrade script to look for this and change the vlan_id. That way new
>>>>>> > installs and upgrades have consistent data. We could fix it in the code
>>>>>> by
>>>>>> > filtering it through
>>>>>> BroadcastDomainType.Vlan.toUri(ipAddr.getVlanTag()),
>>>>>> > but then later when someone expects the vlan id to be in URI format
>>>>>> > somewhere else it will randomly break on people who did an upgrade. The
>>>>>> fix
>>>>>> > to CLOUDSTACK-5505 did this, avoiding the mismatch in format in the DB
>>>>>> by
>>>>>> > fetching the data from elsewhere that filtered it into URI format, and
>>>>>> then
>>>>>> > we just ended up hitting it again here.
>>>>>> >
>>>>>> >
>>>>>> > On Fri, May 30, 2014 at 9:57 AM, Marcus <shadowsor@gmail.com> wrote:
>>>>>> >
>>>>>> >> Actually, if you're in the position to play a bit...  New deployments
>>>>>> seem
>>>>>> >> to work, and I believe it's because that broadcastUri is stored in the
>>>>>> db
>>>>>> >> in the new format:
>>>>>> >>
>>>>>> >> mysql> select id,vlan_id from vlan where network_id = (select id from
>>>>>> >> networks where traffic_type="Public");
>>>>>> >> +----+-----------------+
>>>>>> >> | id | vlan_id         |
>>>>>> >> +----+-----------------+
>>>>>> >> |  1 | vlan://untagged |
>>>>>> >> +----+-----------------+
>>>>>> >> 1 row in set (0.00 sec)
>>>>>> >>
>>>>>> >>
>>>>>> >> I believe 4.2 and earlier would say just 'untagged' there. If you want
>>>>>> to
>>>>>> >> attempt changing that value to include vlan:// (if it is in fact
>>>>>> missing),
>>>>>> >> then restarting everything, that may fix the  issue.
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Fri, May 30, 2014 at 9:38 AM, Marcus <shadowsor@gmail.com> wrote:
>>>>>> >>
>>>>>> >>> I thinnk the commit that caused the change was this or related to it.
>>>>>> Is
>>>>>> >>> there any way you could test a fix? Do you need me to build 4.3 RPMs
>>>>>> or can
>>>>>> >>> I just provide a patch? What works for you?
>>>>>> >>>
>>>>>> >>> commit 53d09c6f1843f04c5f1ab76be9419f5584302d1e
>>>>>> >>> Date:   Mon Aug 5 11:52:40 2013 +0200
>>>>>> >>>
>>>>>> >>>     uri code per broadcast/isolation type , default is to accept
>>>>>> anything
>>>>>> >>> as uri , vlan and lswitch need some extra tlc
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Fri, May 30, 2014 at 9:36 AM, Marcus <shadowsor@gmail.com> wrote:
>>>>>> >>>
>>>>>> >>>> Note the differences in broadcastUri, here is your plug command:
>>>>>> >>>>
>>>>>> >>>> {
>>>>>> >>>>     "com.cloud.agent.api.PlugNicCommand": {
>>>>>> >>>>         "nic": {
>>>>>> >>>>             "deviceId": 1,
>>>>>> >>>>             "networkRateMbps": 99999,
>>>>>> >>>>             "defaultNic": true,
>>>>>> >>>>             "uuid": "6c782af3-2071-4543-acdc-cb30096e89ff",
>>>>>> >>>>             "ip": "46.232.xxx.246",
>>>>>> >>>>             "netmask": "255.255.255.0",
>>>>>> >>>>             "gateway": "46.232.xxx.1",
>>>>>> >>>>             "mac": "06:53:82:00:00:25",
>>>>>> >>>>             "broadcastType": "Vlan",
>>>>>> >>>>             "type": "Public",
>>>>>> >>>>             "broadcastUri": "vlan://untagged",
>>>>>> >>>>             "isolationUri": "vlan://untagged",
>>>>>> >>>>             "isSecurityGroupEnabled": false,
>>>>>> >>>>             "name": "breth1-500"
>>>>>> >>>>         },
>>>>>> >>>>         "instanceName": "r-801-VM",
>>>>>> >>>>         "vmType": "DomainRouter",
>>>>>> >>>>         "wait": 0
>>>>>> >>>>     }
>>>>>> >>>> }
>>>>>> >>>>
>>>>>> >>>> and here is your ip associate command:
>>>>>> >>>>
>>>>>> >>>> {
>>>>>> >>>>     "com.cloud.agent.api.routing.IpAssocVpcCommand": {
>>>>>> >>>>         "ipAddresses": [
>>>>>> >>>>             {
>>>>>> >>>>                 "accountId": 11,
>>>>>> >>>>                 "publicIp": "46.232.xxx.246",
>>>>>> >>>>                 "sourceNat": true,
>>>>>> >>>>                 "add": true,
>>>>>> >>>>                 "oneToOneNat": false,
>>>>>> >>>>                 "firstIP": false,
>>>>>> >>>>                 "broadcastUri": "untagged",
>>>>>> >>>>                 "vlanGateway": "46.232.xxx.1",
>>>>>> >>>>                 "vlanNetmask": "255.255.255.0",
>>>>>> >>>>                 "vifMacAddress": "06:53:82:00:00:25",
>>>>>> >>>>                 "networkRate": 99999,
>>>>>> >>>>                  "trafficType": "Public",
>>>>>> >>>>                 "networkName": "breth1-500"
>>>>>> >>>>             }
>>>>>> >>>>         ],
>>>>>> >>>>         "accessDetails": {
>>>>>> >>>>             "router.guest.ip": "46.232.xxx.246",
>>>>>> >>>>             "zone.network.type": "Advanced",
>>>>>> >>>>             "router.name": "r-801-VM",
>>>>>> >>>>             "router.ip": "169.254.0.52"
>>>>>> >>>>         },
>>>>>> >>>>         "wait": 0
>>>>>> >>>>     }
>>>>>> >>>> }
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> On Fri, May 30, 2014 at 9:27 AM, Andrija Panic <
>>>>>> andrija.panic@gmail.com>
>>>>>> >>>> wrote:
>>>>>> >>>>
>>>>>> >>>>> yes, correct, eth1 is present, and can be started by static IP
>>>>>> >>>>> configuration...
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> On 30 May 2014 17:25, Marcus <shadowsor@gmail.com> wrote:
>>>>>> >>>>>
>>>>>> >>>>> > Let me make sure I understand... the 'Plug' of the nic works
>>>>>> fine, as
>>>>>> >>>>> it
>>>>>> >>>>> > seems you do have an eth1 and can manually assign the IP to get
>>>>>> it to
>>>>>> >>>>> work?
>>>>>> >>>>> > If that's the case then there's probably not an issue in
>>>>>> >>>>> BridgeVifDriver or
>>>>>> >>>>> > the XML. It is definitely in fetching/matching the eth device here
>>>>>> >>>>> >
>>>>>> >>>>> > vpc_ipassoc.sh 169.254.0.52  -A  -l 46.232.xxx.246 -c ethnull -g
>>>>>> >>>>> > 46.232.xxx.1 -m 24 -n 46.232.xxx.0
>>>>>> >>>>> >
>>>>>> >>>>> >
>>>>>> >>>>> > On Fri, May 30, 2014 at 8:33 AM, Andrija Panic <
>>>>>> >>>>> andrija.panic@gmail.com>
>>>>>> >>>>> > wrote:
>>>>>> >>>>> >
>>>>>> >>>>> > > Nope, started, did check, it is reported now as Debian 5 VM, but
>>>>>> >>>>> still
>>>>>> >>>>> > > doesn't work...
>>>>>> >>>>> > > Rebooted VPC (destroyed VR, new one created...)
>>>>>> >>>>> > >
>>>>>> >>>>> > > $ virsh dumpxml r-812-VM
>>>>>> >>>>> > >  ...
>>>>>> >>>>> > > <description>Debian GNU/Linux 5(64-bit)</description>
>>>>>> >>>>> > > ...
>>>>>> >>>>> > > <os>
>>>>>> >>>>> > >     <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
>>>>>> >>>>> > >     <boot dev='cdrom'/>
>>>>>> >>>>> > >     <boot dev='hd'/>
>>>>>> >>>>> > >   </os>
>>>>>> >>>>> > >
>>>>>> >>>>> > > :(
>>>>>> >>>>> > >
>>>>>> >>>>> > >
>>>>>> >>>>> > >
>>>>>> >>>>> > > On 30 May 2014 16:24, Andrija Panic <andrija.panic@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> > >
>>>>>> >>>>> > > > I confirm, the highest is Debian 5 64bit
>>>>>> >>>>> > > >
>>>>>> >>>>> > > > Per docs
>>>>>> >>>>> > > >
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-2-x-to-4-3
>>>>>> >>>>> > > > , you should use Debian 7.0 64bit as OS type for
>>>>>> system-kvm-4.3
>>>>>> >>>>> > > template...
>>>>>> >>>>> > > > (same for xen and vmware templatest)
>>>>>> >>>>> > > >
>>>>>> >>>>> > > > Will change now DB to point to debian 5, and let you know.
>>>>>> >>>>> > > >
>>>>>> >>>>> > > > This is guest_os_id:          184 |           2 | NULL |
>>>>>> >>>>> > > > 986a0e98-39d6-11e3-8f93-0025904e4412 | Debian GNU/Linux
>>>>>> 7(64-bit)
>>>>>> >>>>> > > >
>>>>>> >>>>> > > >
>>>>>> >>>>> > > > On 30 May 2014 15:45, Joris van Lieshout <
>>>>>> >>>>> > > JvanLieshout@schubergphilis.com>
>>>>>> >>>>> > > > wrote:
>>>>>> >>>>> > > >
>>>>>> >>>>> > > >> Andrija,
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> The thing is I don't know who the os matching on KVM works.
>>>>>> >>>>> There must
>>>>>> >>>>> > > be
>>>>>> >>>>> > > >> a way to list supported os types.
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> I also did some queuing on the guest_os_hypervisor table (ACS
>>>>>> >>>>> 4.3)
>>>>>> >>>>> > and I
>>>>>> >>>>> > > >> don't see Debian 7 for KVM listed.
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> select * from guest_os join guest_os_hypervisor on
>>>>>> >>>>> > > >> guest_os.id=guest_os_hypervisor.guest_os_id where
>>>>>> >>>>> > > >> guest_os_hypervisor.hypervisor_type='KVM' and
>>>>>> >>>>> > > >> guest_os_hypervisor.guest_os_name like '%Debian%';
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> What was the guest_os_id you where using? Could you try id 72
>>>>>> >>>>> (Debian
>>>>>> >>>>> > 5
>>>>>> >>>>> > > >> 64-bit)? Adjust both os_type_id in vm_instance and
>>>>>> vm_template
>>>>>> >>>>> (where
>>>>>> >>>>> > > >> type='SYSTEM' and hypervisor_type='KVM').
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> Kind regards,
>>>>>> >>>>> > > >> Joris van Lieshout
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> Schuberg Philis
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> On 30/05/14 15:33, "Andrija Panic" <andrija.panic@gmail.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >> >Joris,
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >do you have recommendation on how in particular to try ? I'm
>>>>>> >>>>> not sure
>>>>>> >>>>> > > how
>>>>>> >>>>> > > >> >to fix that, except playing with editing systemvm-4.3
>>>>>> template
>>>>>> >>>>> to
>>>>>> >>>>> > > define
>>>>>> >>>>> > > >> >it
>>>>>> >>>>> > > >> >as another OS type... ?
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >Thanks again,
>>>>>> >>>>> > > >> >Andrija
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >On 30 May 2014 15:30, Joris van Lieshout <
>>>>>> >>>>> > > >> JvanLieshout@schubergphilis.com>
>>>>>> >>>>> > > >> >wrote:
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >> I've read back a bit in the code and if you look at
>>>>>> >>>>> > > >> BridgeVifDriver.java
>>>>>> >>>>> > > >> >> (this is where the log message with the nic profile is
>>>>>> >>>>> generated)
>>>>>> >>>>> > you
>>>>>> >>>>> > > >> >>can
>>>>>> >>>>> > > >> >> see that the nic information might be off already once ACS
>>>>>> >>>>> hits the
>>>>>> >>>>> > > >> >> LibvirtVMDef.InterfaceDef plug function. This leads be to
>>>>>> >>>>> believer
>>>>>> >>>>> > > that
>>>>>> >>>>> > > >> >> the HVM/PV OS mismatch issue might still be related. Try
>>>>>> >>>>> fixing
>>>>>> >>>>> > that
>>>>>> >>>>> > > >> >> first. At least it will allow us to exclude this from the
>>>>>> >>>>> list.
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >> Kind regards,
>>>>>> >>>>> > > >> >> Joris van Lieshout
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >> Schuberg Philis
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >> On 30/05/14 15:26, "Andrija Panic" <
>>>>>> andrija.panic@gmail.com>
>>>>>> >>>>> > wrote:
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >> >OK, thanks Joris.
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >I will try playing with OS version option, on the
>>>>>> >>>>> systemvm-kvm-4.3
>>>>>> >>>>> > > >> >> >template...
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >Let me know if I can help with anything more.
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >Thanks.
>>>>>> >>>>> > > >> >> >Andrija
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >On 30 May 2014 15:19, Joris van Lieshout
>>>>>> >>>>> > > >> >><JvanLieshout@schubergphilis.com
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >wrote:
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >> Hi Andrija,
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> That does sound familiar and in the start xml of KVM
>>>>>> you
>>>>>> >>>>> can see
>>>>>> >>>>> > > >> >>"<type
>>>>>> >>>>> > > >> >> >> arch='x86_64' machine='pc'>hvm</type>". I don't know
>>>>>> >>>>> KVM+ACS
>>>>>> >>>>> > well
>>>>>> >>>>> > > >> >>enough
>>>>>> >>>>> > > >> >> >> to judge if this is the cause but I thing focusing on
>>>>>> >>>>> getting
>>>>>> >>>>> > the
>>>>>> >>>>> > > VR
>>>>>> >>>>> > > >> >> >> started as PV guest might be worth trying. On the other
>>>>>> >>>>> hand I
>>>>>> >>>>> > do
>>>>>> >>>>> > > >> see
>>>>>> >>>>> > > >> >> >> patchviasocket.pl being executed successfully...
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> The other thing I see is, and now we're getting into
>>>>>> java
>>>>>> >>>>> code,
>>>>>> >>>>> > is
>>>>>> >>>>> > > >> >>this:
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> 2014-05-30 14:41:01,386{GMT} DEBUG
>>>>>> >>>>> > [kvm.resource.BridgeVifDriver]
>>>>>> >>>>> > > >> >> >> (agentRequest-Handler-3:)
>>>>>> >>>>> > > >> >> >>nic=[Nic:Public-46.232.xxx.246-vlan://untagged]
>>>>>> >>>>> > > >> >> >> 2014-05-30 14:41:01,502{GMT} DEBUG [cloud.agent.Agent]
>>>>>> >>>>> > > >> >> >> (agentRequest-Handler-3:) Processing command:
>>>>>> >>>>> > > >> >> >> com.cloud.agent.api.routing.IpAssocVpcCommand
>>>>>> >>>>> > > >> >> >> 2014-05-30 14:41:01,506{GMT} DEBUG
>>>>>> >>>>> > > >> >> >> [resource.virtualnetwork.VirtualRoutingResource]
>>>>>> >>>>> > > >> >> >>(agentRequest-Handler-3:)
>>>>>> >>>>> > > >> >> >> Executing:
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>>>>> >>>>> > > >> >> >> vpc_ipassoc.sh 169.254.0.52  -A  -l 46.232.xxx.246 -c
>>>>>> >>>>> ethnull -g
>>>>>> >>>>> > > >> >> >> 46.232.xxx.1 -m 24 -n 46.232.xxx.0
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> My suspicion is that somewhere in the translation from
>>>>>> the
>>>>>> >>>>> nic
>>>>>> >>>>> > > >> >>profile
>>>>>> >>>>> > > >> >> >>to
>>>>>> >>>>> > > >> >> >> the actual route_proxy.sh command ACS failes to find
>>>>>> the
>>>>>> >>>>> nic id
>>>>>> >>>>> > > and
>>>>>> >>>>> > > >> >> >> returns null.
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> Let me dig a bit deeper and see what I can find but
>>>>>> this is
>>>>>> >>>>> > where
>>>>>> >>>>> > > we
>>>>>> >>>>> > > >> >> >>might
>>>>>> >>>>> > > >> >> >> need some help from someone with knowledge of this
>>>>>> pice of
>>>>>> >>>>> the
>>>>>> >>>>> > > code.
>>>>>> >>>>> > > >> >>:)
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> Kind regards,
>>>>>> >>>>> > > >> >> >> Joris van Lieshout
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> Schuberg Philis
>>>>>> >>>>> > > >> >> >> Boeingavenue 271
>>>>>> >>>>> > > >> >> >> 1119 PD  Schiphol-Rijk
>>>>>> >>>>> > > >> >> >> schubergphilis.com
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> +31 20-7506672
>>>>>> >>>>> > > >> >> >> +31 6-51428188
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> On 30/05/14 14:49, "Andrija Panic" <
>>>>>> >>>>> andrija.panic@gmail.com>
>>>>>> >>>>> > > wrote:
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >> >Hi Joris,
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >I have turned on DEBUG loging in agent.log on
>>>>>> cs1.xxx/net
>>>>>> >>>>> host:
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >So, management logs again:
>>>>>> http://pastebin.com/F6BRf7Y9
>>>>>> >>>>> > > >> >> >> >Agent logs on cs1.xxx: http://pastebin.com/BJauKbaC
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >Not playing smart, but there is some error:
>>>>>> >>>>> > > >> >> >> >[kvm.resource.KVMGuestOsMapper]
>>>>>> >>>>> > > >> >> >> >(agentRequest-Handler-3:) Can't find the mapping of
>>>>>> guest
>>>>>> >>>>> os:
>>>>>> >>>>> > > >> Debian
>>>>>> >>>>> > > >> >> >> >GNU/Linux 7(64-bit)
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >Best,
>>>>>> >>>>> > > >> >> >> >Andrija
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >On 30 May 2014 14:26, Joris van Lieshout
>>>>>> >>>>> > > >> >> >><JvanLieshout@schubergphilis.com
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >wrote:
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >> Hi Andrija,
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> Bold formatting does not come trough on the dev
>>>>>> list. :)
>>>>>> >>>>> > > >> >> >> >> But u might need a bit more info.
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> At a certain point I see this line
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> 2014-05-30 13:56:23,935 DEBUG [c.c.a.t.Request]
>>>>>> >>>>> > > >> >> >> >> (Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
>>>>>> >>>>> 1-609104082:
>>>>>> >>>>> > > >> >>Sending
>>>>>> >>>>> > > >> >> >> {
>>>>>> >>>>> > > >> >> >> >> Cmd , MgmtId: 161344838950, via: 1(cs1.xxxxx.net),
>>>>>> >>>>> Ver: v1,
>>>>>> >>>>> > > >> Flags:
>>>>>> >>>>> > > >> >> >> >>100111,
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>[{"com.cloud.agent.api.StartCommand":{"vm":{"id":801,"name":"r-801-VM
>>>>>> >>>>> > > >> >>>>>>".
>>>>>> >>>>> > > >> >> >>>>..
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> This is where the information is passed on to the
>>>>>> agent
>>>>>> >>>>> > > handles.
>>>>>> >>>>> > > >> >>For
>>>>>> >>>>> > > >> >> >>XS
>>>>>> >>>>> > > >> >> >> >> this would initiate an agent handler on the
>>>>>> management
>>>>>> >>>>> server
>>>>>> >>>>> > > but
>>>>>> >>>>> > > >> >>for
>>>>>> >>>>> > > >> >> >> >>KVM,
>>>>>> >>>>> > > >> >> >> >> if I remember correctly, it passed the command on
>>>>>> to the
>>>>>> >>>>> > > >> >>cloudstack
>>>>>> >>>>> > > >> >> >> >>agent
>>>>>> >>>>> > > >> >> >> >> service on the hypervisor.
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> Can you check the cloud service log on the KVM
>>>>>> >>>>> hypervisor
>>>>>> >>>>> > > >> >>executing
>>>>>> >>>>> > > >> >> >>the
>>>>>> >>>>> > > >> >> >> >> request? it's this server cs1.xxxxx.net and then
>>>>>> >>>>> search top
>>>>>> >>>>> > > down
>>>>>> >>>>> > > >> >>for
>>>>>> >>>>> > > >> >> >> >> 609104082 in the log. See if you can provide the log
>>>>>> >>>>> from the
>>>>>> >>>>> > > >> >>agent
>>>>>> >>>>> > > >> >> >> >> handler thread started by that sequence.
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> Kind regards,
>>>>>> >>>>> > > >> >> >> >> Joris van Lieshout
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> Schuberg Philis
>>>>>> >>>>> > > >> >> >> >> Boeingavenue 271
>>>>>> >>>>> > > >> >> >> >> 1119 PD  Schiphol-Rijk
>>>>>> >>>>> > > >> >> >> >> schubergphilis.com
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> +31 20-7506672
>>>>>> >>>>> > > >> >> >> >> +31 6-51428188
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> On 30/05/14 14:08, "Andrija Panic" <
>>>>>> >>>>> andrija.panic@gmail.com>
>>>>>> >>>>> > > >> >>wrote:
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >Hi Joris,
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >here is the management log:
>>>>>> >>>>> http://pastebin.com/zxnKxFhk
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >Interesting parts (to me): in bold
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:21,899 DEBUG
>>>>>> >>>>> > > >> >>[o.a.c.s.m.AncientDataMotionStrategy]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af)
>>>>>> copyAsync
>>>>>> >>>>> > > inspecting
>>>>>> >>>>> > > >> >>src
>>>>>> >>>>> > > >> >> >> >>type
>>>>>> >>>>> > > >> >> >> >> >TEMPLATE copyAsync inspecting dest type VOLUME
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:21,905 DEBUG [c.c.a.t.Request]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
>>>>>> >>>>> > 4-1248669612:
>>>>>> >>>>> > > >> >> >>Sending
>>>>>> >>>>> > > >> >> >> >>{
>>>>>> >>>>> > > >> >> >> >> >Cmd , MgmtId: 161344838950, via: 4(cs2.xxxxx.net),
>>>>>> >>>>> Ver: v1,
>>>>>> >>>>> > > >> >>Flags:
>>>>>> >>>>> > > >> >> >> >> 100011,
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org
>>>>>> >>>>> > > >> >>>>>>>.a
>>>>>> >>>>> > > >> >> >>>>>pa
>>>>>> >>>>> > > >> >> >> >>>ch
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>e.cloudstack.storage.to.TemplateObjectTO":{"path":"1adc1d2e-56ae-4a0
>>>>>> >>>>> > > >> >>>>>>>f-
>>>>>> >>>>> > > >> >> >>>>>b0
>>>>>> >>>>> > > >> >> >> >>>b4
>>>>>> >>>>> > > >> >> >> >> >-5e351e7cae55","origUrl":"
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-mas
>>>>>> >>>>> > > >> >> >> >>t
>>>>>> >>>>> > > >> >> >> >> >er-kvm.qcow2.bz2
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>","uuid":"1adc1d2e-56ae-4a0f-b0b4-5e351e7cae55","id":414,"format":"Q
>>>>>> >>>>> > > >> >>>>>>>CO
>>>>>> >>>>> > > >> >> >>>>>W2
>>>>>> >>>>> > > >> >> >> >>>",
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > >"accountId":2,"checksum":"85a1bed07bf43cbf022451cb2ecae4ff","
>>>>>> >>>>> > > >> >> >> >> >*hvm":true*
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>,"displayText":"systemvm-kvm-4.3","imageDataStore":{"org.apache.clou
>>>>>> >>>>> > > >> >>>>>>>ds
>>>>>> >>>>> > > >> >> >>>>>ta
>>>>>> >>>>> > > >> >> >> >>>ck
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>.storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-88a8-220
>>>>>> >>>>> > > >> >>>>>>>3f
>>>>>> >>>>> > > >> >> >>>>>79
>>>>>> >>>>> > > >> >> >> >>>b1
>>>>>> >>>>> > > >> >> >> >> >dc6","id":209,"poolType":"RBD","host":"
>>>>>> >>>>> > > >> >> >> >> >cephmon.xxxxx.net
>>>>>> >>>>> > > >> ","path":"cloudstack","port":6789,"url":"RBD://
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8
>>>>>> >>>>> > > >> >> >> >>8
>>>>>> >>>>> > > >> >> >> >> >a8-2203f79b1dc6
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>"}},"name":"414-2-ec331e74-5858-3153-91a9-1d706d9c533e","hypervisorT
>>>>>> >>>>> > > >> >>>>>>>yp
>>>>>> >>>>> > > >> >> >>>>>e"
>>>>>> >>>>> > > >> >> >> >>>:"
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{
>>>>>> >>>>> > > >> >>>>>>>"u
>>>>>> >>>>> > > >> >> >>>>>ui
>>>>>> >>>>> > > >> >> >> >>>d"
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>:"9c440d3b-cba5-4960-b8bf-dca90291cd2b","volumeType":"ROOT","dataSto
>>>>>> >>>>> > > >> >>>>>>>re
>>>>>> >>>>> > > >> >> >>>>>":
>>>>>> >>>>> > > >> >> >> >>>{"
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"5b9342
>>>>>> >>>>> > > >> >>>>>>>2e
>>>>>> >>>>> > > >> >> >>>>>-1
>>>>>> >>>>> > > >> >> >> >>>a6
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> >6-353d-88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>>>>> >>>>> > > >> >> >> >> >cephmon.xxxxx.net
>>>>>> >>>>> > > >> ","path":"cloudstack","port":6789,"url":"RBD://
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> cephmon.xxxxx.net/cloudstack/?ROLE=Primary&STOREUUID=5b93422e-1a66-353d-8
>>>>>> >>>>> > > >> >> >> >>8
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>a8-2203f79b1dc6"}},"name":"ROOT-801","size":2621440000,"volumeId":10
>>>>>> >>>>> > > >> >>>>>>>64
>>>>>> >>>>> > > >> >> >>>>>,"
>>>>>> >>>>> > > >> >> >> >>>vm
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>Name":"r-801-VM","accountId":11,"format":"RAW","id":1064,"deviceId":
>>>>>> >>>>> > > >> >>>>>>>0,
>>>>>> >>>>> > > >> >> >>>>>"h
>>>>>> >>>>> > > >> >> >> >>>yp
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>ervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}
>>>>>> >>>>> > > >> >>>>>]
>>>>>> >>>>> > > >> >> >> >> >}
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
>>>>>> >>>>> > > >> >> >> >> >(AgentManager-Handler-12:null) Seq 4-1248669612:
>>>>>> >>>>> Processing:
>>>>>> >>>>> > >  {
>>>>>> >>>>> > > >> >> >>Ans: ,
>>>>>> >>>>> > > >> >> >> >> >MgmtId: 161344838950, via: 4, Ver: v1, Flags: 10,
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{
>>>>>> >>>>> > > >> >>>>>>>"o
>>>>>> >>>>> > > >> >> >>>>>rg
>>>>>> >>>>> > > >> >> >> >>>.a
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>pache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path
>>>>>> >>>>> > > >> >>>>>>>":
>>>>>> >>>>> > > >> >> >>>>>"9
>>>>>> >>>>> > > >> >> >> >>>c4
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>40d3b-cba5-4960-b8bf-dca90291cd2b","accountId":0,"format":"RAW","id"
>>>>>> >>>>> > > >> >>>>>>>:0
>>>>>> >>>>> > > >> >> >>>>>}}
>>>>>> >>>>> > > >> >> >> >>>,"
>>>>>> >>>>> > > >> >> >> >> >result":true,"wait":0}}]
>>>>>> >>>>> > > >> >> >> >> >}
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:23,742 DEBUG [c.c.a.t.Request]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af) Seq
>>>>>> >>>>> > 4-1248669612:
>>>>>> >>>>> > > >> >> >> >>Received:  {
>>>>>> >>>>> > > >> >> >> >> >Ans: , MgmtId: 161344838950, via: 4, Ver: v1,
>>>>>> Flags:
>>>>>> >>>>> 10, {
>>>>>> >>>>> > > >> >> >> >>CopyCmdAnswer
>>>>>> >>>>> > > >> >> >> >> >} }
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>>> >>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af)
>>>>>> Removing
>>>>>> >>>>> nic
>>>>>> >>>>> > > >> >>NicProfile[
>>>>>> >>>>> > > >> >> >> >> >*1092-801-null*-46.232.xxx.246-vlan://untagged of
>>>>>> type
>>>>>> >>>>> > Public
>>>>>> >>>>> > > >> >>from
>>>>>> >>>>> > > >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >nics
>>>>>> >>>>> > > >> >> >> >> >passed on vm start. The nic will be plugged later
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>>> >>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af)
>>>>>> Removing
>>>>>> >>>>> nic
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>NicProfile[1093-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.1.1-vl
>>>>>> >>>>> > > >> >>>>>>>an
>>>>>> >>>>> > > >> >> >>>>>:/
>>>>>> >>>>> > > >> >> >> >>>/4
>>>>>> >>>>> > > >> >> >> >> >4
>>>>>> >>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
>>>>>> >>>>> nic will
>>>>>> >>>>> > > be
>>>>>> >>>>> > > >> >> >>plugged
>>>>>> >>>>> > > >> >> >> >> >later
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>>> >>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af)
>>>>>> Removing
>>>>>> >>>>> nic
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>NicProfile[1094-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.3.1-vl
>>>>>> >>>>> > > >> >>>>>>>an
>>>>>> >>>>> > > >> >> >>>>>:/
>>>>>> >>>>> > > >> >> >> >>>/4
>>>>>> >>>>> > > >> >> >> >> >3
>>>>>> >>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
>>>>>> >>>>> nic will
>>>>>> >>>>> > > be
>>>>>> >>>>> > > >> >> >>plugged
>>>>>> >>>>> > > >> >> >> >> >later
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>>> >>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af)
>>>>>> Removing
>>>>>> >>>>> nic
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl
>>>>>> >>>>> > > >> >>>>>>>an
>>>>>> >>>>> > > >> >> >>>>>:/
>>>>>> >>>>> > > >> >> >> >>>/3
>>>>>> >>>>> > > >> >> >> >> >004
>>>>>> >>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start. The
>>>>>> >>>>> nic will
>>>>>> >>>>> > > be
>>>>>> >>>>> > > >> >> >>plugged
>>>>>> >>>>> > > >> >> >> >> >later
>>>>>> >>>>> > > >> >> >> >> >2014-05-30 13:56:23,773 DEBUG
>>>>>> >>>>> > > >> >> >> >> >[c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
>>>>>> >>>>> > > >> >> >> >> >(Job-Executor-77:ctx-ec3d358e ctx-f35b12af)
>>>>>> Removing
>>>>>> >>>>> nic
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>>>NicProfile[1095-801-cd9fd29a-0573-4715-8742-00ecb9f82c9d-10.0.4.1-vl
>>>>>> >>>>> > > >> >>>>>>>an
>>>>>> >>>>> > > >> >> >>>>>:/
>>>>>> >>>>> > > >> >> >> >>>/3
>>>>>> >>>>> > > >> >> >> >> >004
>>>>>> >>>>> > > >> >> >> >> >of type Guest from the nics passed on vm start
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >Thanks,
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >On 30 May 2014 13:54, Joris van Lieshout
>>>>>> >>>>> > > >> >> >> >><JvanLieshout@schubergphilis.com
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >wrote:
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> Hi Andrija,
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> Just the start of the VR should be sufficient.
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> Kind regards,
>>>>>> >>>>> > > >> >> >> >> >> Joris van Lieshout
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> Schuberg Philis
>>>>>> >>>>> > > >> >> >> >> >> Boeingavenue 271
>>>>>> >>>>> > > >> >> >> >> >> 1119 PD  Schiphol-Rijk
>>>>>> >>>>> > > >> >> >> >> >> schubergphilis.com
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> +31 20-7506672
>>>>>> >>>>> > > >> >> >> >> >> +31 6-51428188
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> On 30/05/14 13:48, "Andrija Panic" <
>>>>>> >>>>> > andrija.panic@gmail.com
>>>>>> >>>>> > > >
>>>>>> >>>>> > > >> >> >>wrote:
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >Hi Joris,
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >just to be sure - you want me to capture the log
>>>>>> >>>>> from the
>>>>>> >>>>> > > >> >>moment
>>>>>> >>>>> > > >> >> >>I
>>>>>> >>>>> > > >> >> >> >> >>reboot
>>>>>> >>>>> > > >> >> >> >> >> >router - or you want me to stop it, then start
>>>>>> >>>>> capturing
>>>>>> >>>>> > > log,
>>>>>> >>>>> > > >> >>and
>>>>>> >>>>> > > >> >> >> >> >>start it
>>>>>> >>>>> > > >> >> >> >> >> >(and continue capture untill ethnull errors
>>>>>> inside
>>>>>> >>>>> VR) ?
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >Thanks,
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >On 30 May 2014 13:39, Joris van Lieshout
>>>>>> >>>>> > > >> >> >> >> >><JvanLieshout@schubergphilis.com
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >wrote:
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> Hi Andrija,
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> Thanks for the answers. In deed your
>>>>>> situation is
>>>>>> >>>>> > > different
>>>>>> >>>>> > > >> >>so
>>>>>> >>>>> > > >> >> >> >> >>PV/HVM is
>>>>>> >>>>> > > >> >> >> >> >> >> not the issue.
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> When reading back the log output you have
>>>>>> >>>>> provided I
>>>>>> >>>>> > > noted
>>>>>> >>>>> > > >> >>that
>>>>>> >>>>> > > >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >>VR
>>>>>> >>>>> > > >> >> >> >> >> >> messages log indicates that it's waiting for
>>>>>> >>>>> ethnull to
>>>>>> >>>>> > > be
>>>>>> >>>>> > > >> >>up.
>>>>>> >>>>> > > >> >> >> >>This
>>>>>> >>>>> > > >> >> >> >> >> >>raises
>>>>>> >>>>> > > >> >> >> >> >> >> the question where null was introduced
>>>>>> instead of
>>>>>> >>>>> 1.
>>>>>> >>>>> > The
>>>>>> >>>>> > > >> ACS
>>>>>> >>>>> > > >> >> >> >> >>management
>>>>>> >>>>> > > >> >> >> >> >> >> log output you send was, what I think, later
>>>>>> down
>>>>>> >>>>> the
>>>>>> >>>>> > > road
>>>>>> >>>>> > > >> >> >>where
>>>>>> >>>>> > > >> >> >> >>ACS
>>>>>> >>>>> > > >> >> >> >> >> >>gives
>>>>>> >>>>> > > >> >> >> >> >> >> up trying to wait for the VR to come up. If
>>>>>> you
>>>>>> >>>>> would
>>>>>> >>>>> > > >> >>capture
>>>>>> >>>>> > > >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> job-executor in the management log from
>>>>>> >>>>> startCommand
>>>>>> >>>>> > till
>>>>>> >>>>> > > >> >>the
>>>>>> >>>>> > > >> >> >> >> >>exception,
>>>>>> >>>>> > > >> >> >> >> >> >> do you see anywhere a mention of ethnull? You
>>>>>> >>>>> might
>>>>>> >>>>> > need
>>>>>> >>>>> > > to
>>>>>> >>>>> > > >> >> >>reed
>>>>>> >>>>> > > >> >> >> >>into
>>>>>> >>>>> > > >> >> >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> DirectAgent executing the startCommand to
>>>>>> find a
>>>>>> >>>>> clue.
>>>>>> >>>>> > > The
>>>>>> >>>>> > > >> >> >>thing
>>>>>> >>>>> > > >> >> >> >>is
>>>>>> >>>>> > > >> >> >> >> >> >>that I
>>>>>> >>>>> > > >> >> >> >> >> >> only have experience with XS based
>>>>>> environment so
>>>>>> >>>>> I
>>>>>> >>>>> > > cannot
>>>>>> >>>>> > > >> >> >>point
>>>>>> >>>>> > > >> >> >> >>you
>>>>>> >>>>> > > >> >> >> >> >>to
>>>>>> >>>>> > > >> >> >> >> >> >> the exact output to look for. On XS, at
>>>>>> least, it
>>>>>> >>>>> is
>>>>>> >>>>> > > >> >> >> >> >> >> "[c.c.h.x.r.CitrixResourceBase]
>>>>>> >>>>> > > >> >>(DirectAgent-351:ctx-4a51bb9e)
>>>>>> >>>>> > > >> >> >> >> >>Created a
>>>>>> >>>>> > > >> >> >> >> >> >> vif e4c362bd-764b-f651-dc9a-1abd5cb33c43 on 1"
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> Kind regards,
>>>>>> >>>>> > > >> >> >> >> >> >> Joris van Lieshout
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> Schuberg Philis
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> On 30/05/14 10:48, "Andrija Panic" <
>>>>>> >>>>> > > >> andrija.panic@gmail.com>
>>>>>> >>>>> > > >> >> >> >>wrote:
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >Hi Deen,
>>>>>> >>>>> > > >> >> >> >> >> >> >no, in DB there is field "vlan_id" with value
>>>>>> >>>>> > > "untagged" -
>>>>>> >>>>> > > >> >> >>that
>>>>>> >>>>> > > >> >> >> >> >> >> >"vlan://untagged" is shown from ACS gui, and
>>>>>> is
>>>>>> >>>>> used
>>>>>> >>>>> > in
>>>>>> >>>>> > > >> API
>>>>>> >>>>> > > >> >> >>call
>>>>>> >>>>> > > >> >> >> >>(or
>>>>>> >>>>> > > >> >> >> >> >> >> >better
>>>>>> >>>>> > > >> >> >> >> >> >> >said commands that are seen in management
>>>>>> server
>>>>>> >>>>> > logs).
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >Best,
>>>>>> >>>>> > > >> >> >> >> >> >> >Andrija
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >On 30 May 2014 10:37, Daan Hoogland
>>>>>> >>>>> > > >> >><daan.hoogland@gmail.com>
>>>>>> >>>>> > > >> >> >> >> wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> Andrija,
>>>>>> >>>>> > > >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> Do not just assign a second net vlan://500
>>>>>> You
>>>>>> >>>>> have
>>>>>> >>>>> > > one
>>>>>> >>>>> > > >> >>like
>>>>>> >>>>> > > >> >> >> >>that
>>>>>> >>>>> > > >> >> >> >> >>and
>>>>>> >>>>> > > >> >> >> >> >> >> >> you don't want conflicting nets using the
>>>>>> same
>>>>>> >>>>> > vlan. I
>>>>>> >>>>> > > >> am
>>>>>> >>>>> > > >> >> >> >> >>wondering
>>>>>> >>>>> > > >> >> >> >> >> >> >> why 'untagged' comes out as
>>>>>> 'vlan://untagged'.
>>>>>> >>>>> I
>>>>>> >>>>> > think
>>>>>> >>>>> > > >> >>that
>>>>>> >>>>> > > >> >> >>is
>>>>>> >>>>> > > >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> >> bug. Did you find the string
>>>>>> 'vlan://untagged'
>>>>>> >>>>> in
>>>>>> >>>>> > your
>>>>>> >>>>> > > >> >>db?
>>>>>> >>>>> > > >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> On Fri, May 30, 2014 at 10:20 AM, Andrija
>>>>>> Panic
>>>>>> >>>>> > > >> >> >> >> >> >> >><andrija.panic@gmail.com>
>>>>>> >>>>> > > >> >> >> >> >> >> >> wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Hi Joris,
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > thank you for taking time to address this
>>>>>> >>>>> issue :)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > So...:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > - I'm on KVM (stock CentOS 6.2 patched by
>>>>>> >>>>> Inktank
>>>>>> >>>>> > > for
>>>>>> >>>>> > > >> >>CEPH
>>>>>> >>>>> > > >> >> >> >> >> >>support),
>>>>>> >>>>> > > >> >> >> >> >> >> >>OS
>>>>>> >>>>> > > >> >> >> >> >> >> >> is
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Centos 6.5, libvirt 1.2.3 compiled.
>>>>>> >>>>> > > >> >> >> >> >> >> >> > - ACS 4.3 having problems, ACS 4.2.1 was
>>>>>> fine
>>>>>> >>>>> > > >> >> >> >> >> >> >> > - not XS, so I guess no answers for this
>>>>>> >>>>> part :)
>>>>>> >>>>> > > >> >> >> >> >> >> >> > - guest_os_id is 184 = Debian 7 x64
>>>>>> >>>>> > > >> >> >> >> >> >> >> > - SVM = systemvm-kvm-4.3 = os type 184 =
>>>>>> >>>>> Debian 7
>>>>>> >>>>> > > x64
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > This worked previously on 4.2.1 =
>>>>>> template
>>>>>> >>>>> was
>>>>>> >>>>> > > >> ofcourse
>>>>>> >>>>> > > >> >> >> >> >> >> >>systemvm-kvm-4.2
>>>>>> >>>>> > > >> >> >> >> >> >> >> -
>>>>>> >>>>> > > >> >> >> >> >> >> >> > but that was also Debian 7 x64 type... so
>>>>>> >>>>> this
>>>>>> >>>>> > > should
>>>>>> >>>>> > > >> >>not
>>>>>> >>>>> > > >> >> >>be
>>>>>> >>>>> > > >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> >>issues
>>>>>> >>>>> > > >> >> >> >> >> >> >> > (guest not supported by host...)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > The only thing that might be out of
>>>>>> >>>>> "standard" =
>>>>>> >>>>> > all
>>>>>> >>>>> > > >> >>SVMs
>>>>>> >>>>> > > >> >> >> >>are on
>>>>>> >>>>> > > >> >> >> >> >> >>CEPH
>>>>>> >>>>> > > >> >> >> >> >> >> >>-
>>>>>> >>>>> > > >> >> >> >> >> >> >> > there are official docs on altering
>>>>>> database
>>>>>> >>>>> to
>>>>>> >>>>> > make
>>>>>> >>>>> > > >> >>some
>>>>>> >>>>> > > >> >> >>new
>>>>>> >>>>> > > >> >> >> >> >> >>System
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Offering as default for SSVM and CPVM -
>>>>>> what
>>>>>> >>>>> I
>>>>>> >>>>> > did,
>>>>>> >>>>> > > I
>>>>>> >>>>> > > >> >>also
>>>>>> >>>>> > > >> >> >> >>have
>>>>>> >>>>> > > >> >> >> >> >> >>done
>>>>>> >>>>> > > >> >> >> >> >> >> >>same
>>>>>> >>>>> > > >> >> >> >> >> >> >> > config in DB, to make VR use another
>>>>>> System
>>>>>> >>>>> > Offering
>>>>>> >>>>> > > >> as
>>>>>> >>>>> > > >> >> >> >>default
>>>>>> >>>>> > > >> >> >> >> >>-
>>>>>> >>>>> > > >> >> >> >> >> >> >>which
>>>>>> >>>>> > > >> >> >> >> >> >> >> is
>>>>>> >>>>> > > >> >> >> >> >> >> >> > NOT explained in the docs - you could use
>>>>>> >>>>> "Change
>>>>>> >>>>> > > >> >> >> >>Offering..."
>>>>>> >>>>> > > >> >> >> >> >> >>button
>>>>>> >>>>> > > >> >> >> >> >> >> >>on
>>>>>> >>>>> > > >> >> >> >> >> >> >> > exiting, shutdown VR to change it per
>>>>>> docs...
>>>>>> >>>>> > > >> >> >> >> >> >> >> > But still this worked all fine on
>>>>>> 4.2.1...
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > - regarding /var/cache/cloud/cmdline the
>>>>>> >>>>> content
>>>>>> >>>>> > is
>>>>>> >>>>> > > >> >> >>folowing
>>>>>> >>>>> > > >> >> >> >>at
>>>>>> >>>>> > > >> >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> >> moment
>>>>>> >>>>> > > >> >> >> >> >> >> >> > root@r-801-VM:~# cat
>>>>>> >>>>> /var/cache/cloud/cmdline
>>>>>> >>>>> > > >> >> >> >> >> >> >> > vpccidr=10.0.0.0/8
>>>>>> domain=cscloud.internal
>>>>>> >>>>> > > >> dns1=8.8.8.8
>>>>>> >>>>> > > >> >> >> dns2=
>>>>>> >>>>> > > >> >> >> >> >> >> >> template=domP
>>>>>> >>>>> > > >> >> >> >> >> >> >> > name=r-801-VM eth0ip=169.254.0.75
>>>>>> >>>>> > > eth0mask=255.255.0.0
>>>>>> >>>>> > > >> >> >> >> >> >>type=vpcrouter
>>>>>> >>>>> > > >> >> >> >> >> >> >> > disable_rp_filter=true
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Also please note that only eth1 does not
>>>>>> >>>>> have IP
>>>>>> >>>>> > > info,
>>>>>> >>>>> > > >> >> >>eth0
>>>>>> >>>>> > > >> >> >> >> >> >>(control
>>>>>> >>>>> > > >> >> >> >> >> >> >> > 169.xxx) and all other eh2 and up that
>>>>>> are
>>>>>> >>>>> used
>>>>>> >>>>> > for
>>>>>> >>>>> > > >> >>Tiers
>>>>>> >>>>> > > >> >> >> >>get IP
>>>>>> >>>>> > > >> >> >> >> >> >>info
>>>>>> >>>>> > > >> >> >> >> >> >> >> fine.
>>>>>> >>>>> > > >> >> >> >> >> >> >> > I could also manually add IP for eth1
>>>>>> >>>>> (public NIC)
>>>>>> >>>>> > > and
>>>>>> >>>>> > > >> >> >>start
>>>>>> >>>>> > > >> >> >> >> >>ifup
>>>>>> >>>>> > > >> >> >> >> >> >> >>eth1 -
>>>>>> >>>>> > > >> >> >> >> >> >> >> > and it works fine, but adding new IP Port
>>>>>> >>>>> > Forwarding
>>>>>> >>>>> > > >> >>etc
>>>>>> >>>>> > > >> >> >>does
>>>>>> >>>>> > > >> >> >> >> >>not
>>>>>> >>>>> > > >> >> >> >> >> >> >>work...
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Daan or somebody said it could be
>>>>>> realted to
>>>>>> >>>>> my
>>>>>> >>>>> > > >> >>"Public"
>>>>>> >>>>> > > >> >> >> >>network
>>>>>> >>>>> > > >> >> >> >> >> >>(in
>>>>>> >>>>> > > >> >> >> >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Zones, Physical Network, eth1 listing)
>>>>>> is NOT
>>>>>> >>>>> > tagged
>>>>>> >>>>> > > >> >> >> >> >> >> >>(vlan://untagged)...
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Interestingly the only VR that does work
>>>>>> >>>>> fine is
>>>>>> >>>>> > the
>>>>>> >>>>> > > >> VR
>>>>>> >>>>> > > >> >> >>used
>>>>>> >>>>> > > >> >> >> >>in
>>>>>> >>>>> > > >> >> >> >> >> >>Shared
>>>>>> >>>>> > > >> >> >> >> >> >> >> > network, but that VR is using IP from
>>>>>> Guest
>>>>>> >>>>> IP
>>>>>> >>>>> > range
>>>>>> >>>>> > > >> >>(also
>>>>>> >>>>> > > >> >> >> >> >> >>efectively
>>>>>> >>>>> > > >> >> >> >> >> >> >> > public IPs but on vlan 500)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > I was instructed to try to change Public
>>>>>> IP
>>>>>> >>>>> range
>>>>>> >>>>> > > from
>>>>>> >>>>> > > >> >> >> >>untagged
>>>>>> >>>>> > > >> >> >> >> >>to
>>>>>> >>>>> > > >> >> >> >> >> >> >>vlan
>>>>>> >>>>> > > >> >> >> >> >> >> >> > 500, but I'm not sure how to do this, if
>>>>>> >>>>> there is
>>>>>> >>>>> > > any
>>>>>> >>>>> > > >> >>way
>>>>>> >>>>> > > >> >> >>at
>>>>>> >>>>> > > >> >> >> >>all
>>>>>> >>>>> > > >> >> >> >> >> >> >>(editing
>>>>>> >>>>> > > >> >> >> >> >> >> >> > "vlan" table and changing to vlan 500
>>>>>> does
>>>>>> >>>>> not
>>>>>> >>>>> > work,
>>>>>> >>>>> > > >> >>after
>>>>>> >>>>> > > >> >> >> >> >> >>rebooting
>>>>>> >>>>> > > >> >> >> >> >> >> >>VR
>>>>>> >>>>> > > >> >> >> >> >> >> >> > from ACS gui).
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > :)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > So, not sure what is roughly expected
>>>>>> date
>>>>>> >>>>> for
>>>>>> >>>>> > 4.4,
>>>>>> >>>>> > > >> but
>>>>>> >>>>> > > >> >> >>right
>>>>>> >>>>> > > >> >> >> >> >>now,
>>>>>> >>>>> > > >> >> >> >> >> >>I'm
>>>>>> >>>>> > > >> >> >> >> >> >> >> > pretty stuck with a big problem of all
>>>>>> VPC
>>>>>> >>>>> not
>>>>>> >>>>> > > >> >> >>operational at
>>>>>> >>>>> > > >> >> >> >> >> >>all...
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Thanks,
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > On 30 May 2014 08:27, Joris van Lieshout
>>>>>> <
>>>>>> >>>>> > > >> >> >> >> >> >> >> JvanLieshout@schubergphilis.com>
>>>>>> >>>>> > > >> >> >> >> >> >> >> > wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Hi Andrija,
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Daan asked me to have a look at this as
>>>>>> >>>>> well.
>>>>>> >>>>> > > Looking
>>>>>> >>>>> > > >> >>at
>>>>>> >>>>> > > >> >> >>you
>>>>>> >>>>> > > >> >> >> >> >> >>issue I
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> recall having seen something similar.
>>>>>> Back
>>>>>> >>>>> then
>>>>>> >>>>> > > when
>>>>>> >>>>> > > >> >> >> >>upgrading
>>>>>> >>>>> > > >> >> >> >> >> >>4.2.1
>>>>>> >>>>> > > >> >> >> >> >> >> >>to
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> 4.3 I though it had to do with out own
>>>>>> >>>>> custom
>>>>>> >>>>> > build
>>>>>> >>>>> > > >> >>svm
>>>>>> >>>>> > > >> >> >> >> >>template.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Let me fire off some questions before
>>>>>> >>>>> explaining
>>>>>> >>>>> > > what
>>>>>> >>>>> > > >> >>the
>>>>>> >>>>> > > >> >> >> >>cause
>>>>>> >>>>> > > >> >> >> >> >> >>was
>>>>>> >>>>> > > >> >> >> >> >> >> >>in
>>>>>> >>>>> > > >> >> >> >> >> >> >> our
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> case. :)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> - what hypervisor (and version) are you
>>>>>> >>>>> using?
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> - if XS, is the new VR a
>>>>>> para-virtualised
>>>>>> >>>>> > instance
>>>>>> >>>>> > > >> >>(PV)
>>>>>> >>>>> > > >> >> >>or
>>>>>> >>>>> > > >> >> >> >> >> >>hardware
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> assisted (HVM)? Do a "xe vm-param-list"
>>>>>> on
>>>>>> >>>>> the VR
>>>>>> >>>>> > > >> uuid
>>>>>> >>>>> > > >> >> >>and
>>>>>> >>>>> > > >> >> >> >> >>check
>>>>>> >>>>> > > >> >> >> >> >> >>that
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> param PV-args is set and
>>>>>> HVM-boot-policy is
>>>>>> >>>>> > unset.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> - what is the OS type of the VR in ACS
>>>>>> >>>>> > (guest_os_id
>>>>>> >>>>> > > >> in
>>>>>> >>>>> > > >> >> >> >> >>vm_instance
>>>>>> >>>>> > > >> >> >> >> >> >> >>table
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> and match with table guest_os)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> - what is the OS type of the SVM
>>>>>> template?
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Now for the explaining. :)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> In our case the OS type of the new
>>>>>> template
>>>>>> >>>>> was
>>>>>> >>>>> > not
>>>>>> >>>>> > > >> >> >> >>supported
>>>>>> >>>>> > > >> >> >> >> >>on
>>>>>> >>>>> > > >> >> >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> XenServer version we are running.
>>>>>> Therefore
>>>>>> >>>>> the
>>>>>> >>>>> > VR
>>>>>> >>>>> > > >> was
>>>>>> >>>>> > > >> >> >> >>started
>>>>>> >>>>> > > >> >> >> >> >>by
>>>>>> >>>>> > > >> >> >> >> >> >>XS
>>>>>> >>>>> > > >> >> >> >> >> >> >>as
>>>>>> >>>>> > > >> >> >> >> >> >> >> a
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> HVM guest. System vms on XS rely on the
>>>>>> >>>>> arguments
>>>>>> >>>>> > > >> >>passed
>>>>>> >>>>> > > >> >> >>to
>>>>>> >>>>> > > >> >> >> >> >>them
>>>>>> >>>>> > > >> >> >> >> >> >>in
>>>>>> >>>>> > > >> >> >> >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> PV-args param (ends up on the guest in
>>>>>> >>>>> > > >> >> >> >>/var/cache/cloud/cmdline
>>>>>> >>>>> > > >> >> >> >> >> >> >>which in
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> turn is used by cloud-early-config) in
>>>>>> >>>>> order to
>>>>>> >>>>> > > work.
>>>>>> >>>>> > > >> >> >> >>cmdline
>>>>>> >>>>> > > >> >> >> >> >> >> >>contains
>>>>>> >>>>> > > >> >> >> >> >> >> >> the
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> NIC configuration information.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> So, long story short, if a VR gets
>>>>>> started
>>>>>> >>>>> as a
>>>>>> >>>>> > HVM
>>>>>> >>>>> > > >> it
>>>>>> >>>>> > > >> >> >>will
>>>>>> >>>>> > > >> >> >> >>not
>>>>>> >>>>> > > >> >> >> >> >> >>get
>>>>>> >>>>> > > >> >> >> >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> information needed to configure it's
>>>>>> NICs.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Workaround
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> We corrected the os_type_id in the DB
>>>>>> (yes
>>>>>> >>>>> I know
>>>>>> >>>>> > > >> >>editing
>>>>>> >>>>> > > >> >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >>DB
>>>>>> >>>>> > > >> >> >> >> >> >>is
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> something you usually don't want but
>>>>>> there
>>>>>> >>>>> is no
>>>>>> >>>>> > > >> other
>>>>>> >>>>> > > >> >> >>way
>>>>>> >>>>> > > >> >> >> >>in
>>>>>> >>>>> > > >> >> >> >> >>this
>>>>>> >>>>> > > >> >> >> >> >> >> >>case)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> of the existing VR's and of the
>>>>>> >>>>> systemvmtemplate
>>>>>> >>>>> > to
>>>>>> >>>>> > > >> >> >> >>something
>>>>>> >>>>> > > >> >> >> >> >> >> >>supported
>>>>>> >>>>> > > >> >> >> >> >> >> >> by
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> XenServer.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Kind regards,
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Joris van Lieshout
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> Schuberg Philis
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> On 29/05/14 12:18, "Andrija Panic"
>>>>>> >>>>> > > >> >> >><andrija.panic@gmail.com
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >>wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >They are 2 traffic types on 1 physical
>>>>>> net
>>>>>> >>>>> (that
>>>>>> >>>>> > > is
>>>>>> >>>>> > > >> >>both
>>>>>> >>>>> > > >> >> >> >> >>tagged
>>>>>> >>>>> > > >> >> >> >> >> >>vlan
>>>>>> >>>>> > > >> >> >> >> >> >> >> 500,
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >and untagged packets travel over same
>>>>>> KVM
>>>>>> >>>>> > bridge,
>>>>>> >>>>> > > >> and
>>>>>> >>>>> > > >> >> >>over
>>>>>> >>>>> > > >> >> >> >> >>eth1
>>>>>> >>>>> > > >> >> >> >> >> >>to
>>>>>> >>>>> > > >> >> >> >> >> >> >> outside
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >world)...
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >On 29 May 2014 12:04, Daan Hoogland
>>>>>> >>>>> > > >> >> >> >><daan.hoogland@gmail.com>
>>>>>> >>>>> > > >> >> >> >> >> >> wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> Are these two traffic types in one
>>>>>> >>>>> physical
>>>>>> >>>>> > net?
>>>>>> >>>>> > > >> or
>>>>>> >>>>> > > >> >> >>two
>>>>>> >>>>> > > >> >> >> >> >> >>physical
>>>>>> >>>>> > > >> >> >> >> >> >> >>nets
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> on the same interface (seems wrong).
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> On Thu, May 29, 2014 at 11:35 AM,
>>>>>> Jayapal
>>>>>> >>>>> > Reddy
>>>>>> >>>>> > > >> >>Uradi
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> <jayapalreddy.uradi@citrix.com>
>>>>>> wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> > I don't think editing DB table will
>>>>>> >>>>> work.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> > -Jayapal
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> > On 29-May-2014, at 2:52 PM, Andrija
>>>>>> >>>>> Panic
>>>>>> >>>>> > > >> >> >> >> >> >> >><andrija.panic@gmail.com
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> It's like this:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> I have public subnet /24.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> half is dedicated for Guest
>>>>>> traffic
>>>>>> >>>>> (vlan
>>>>>> >>>>> > > 500)
>>>>>> >>>>> > > >> >>and
>>>>>> >>>>> > > >> >> >>the
>>>>>> >>>>> > > >> >> >> >> >> >>second
>>>>>> >>>>> > > >> >> >> >> >> >> >> half is
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> dedicated to Public
>>>>>> traffic/network
>>>>>> >>>>> (no
>>>>>> >>>>> > vlan
>>>>>> >>>>> > > >> >>tags,
>>>>>> >>>>> > > >> >> >> >>that
>>>>>> >>>>> > > >> >> >> >> >>is
>>>>>> >>>>> > > >> >> >> >> >> >> >> untagged
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> packets)
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> Both vlan500 and untagged packets
>>>>>> >>>>> travel
>>>>>> >>>>> > over
>>>>>> >>>>> > > >> >> >>physical
>>>>>> >>>>> > > >> >> >> >> >>eth1
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >>interface on
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> hypervisors and can reach
>>>>>> Internet.
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> Thanks,
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> On 29 May 2014 11:06, Daan
>>>>>> Hoogland
>>>>>> >>>>> > > >> >> >> >> >> >><daan.hoogland@gmail.com>
>>>>>> >>>>> > > >> >> >> >> >> >> >> wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>> On Thu, May 29, 2014 at 10:57 AM,
>>>>>> >>>>> Andrija
>>>>>> >>>>> > > >> >>Panic <
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> andrija.panic@gmail.com>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>> wrote:
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>>> 500
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>> is 500 the vlan of your
>>>>>> guestnetwork
>>>>>> >>>>> or
>>>>>> >>>>> > your
>>>>>> >>>>> > > >> >> >>physical
>>>>>> >>>>> > > >> >> >> >> >> >>network?
>>>>>> >>>>> > > >> >> >> >> >> >> >> You
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>> wouldn't want to have two nets
>>>>>> with
>>>>>> >>>>> vlan
>>>>>> >>>>> > > 500!
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>> --
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>> Daan
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> --
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >> Andrija Panić
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> --------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>  http://admintweets.com
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >>
>>>>>> --------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> --
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >> Daan
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >--
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >Andrija Panić
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >  http://admintweets.com
>>>>>> >>>>> > > >> >> >> >> >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > --
>>>>>> >>>>> > > >> >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >> > Andrija Panić
>>>>>> >>>>> > > >> >> >> >> >> >> >> > --------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >> >> >   http://admintweets.com
>>>>>> >>>>> > > >> >> >> >> >> >> >> > --------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >> --
>>>>>> >>>>> > > >> >> >> >> >> >> >> Daan
>>>>>> >>>>> > > >> >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >--
>>>>>> >>>>> > > >> >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >> >Andrija Panić
>>>>>> >>>>> > > >> >> >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >> >  http://admintweets.com
>>>>>> >>>>> > > >> >> >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >--
>>>>>> >>>>> > > >> >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >> >Andrija Panić
>>>>>> >>>>> > > >> >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >> >> >  http://admintweets.com
>>>>>> >>>>> > > >> >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >>
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >--
>>>>>> >>>>> > > >> >> >> >> >
>>>>>> >>>>> > > >> >> >> >> >Andrija Panić
>>>>>> >>>>> > > >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >> >  http://admintweets.com
>>>>>> >>>>> > > >> >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >>
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >--
>>>>>> >>>>> > > >> >> >> >
>>>>>> >>>>> > > >> >> >> >Andrija Panić
>>>>>> >>>>> > > >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >> >  http://admintweets.com
>>>>>> >>>>> > > >> >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >>
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >--
>>>>>> >>>>> > > >> >> >
>>>>>> >>>>> > > >> >> >Andrija Panić
>>>>>> >>>>> > > >> >> >--------------------------------------
>>>>>> >>>>> > > >> >> >  http://admintweets.com
>>>>>> >>>>> > > >> >> >--------------------------------------
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >>
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >--
>>>>>> >>>>> > > >> >
>>>>>> >>>>> > > >> >Andrija Panić
>>>>>> >>>>> > > >> >--------------------------------------
>>>>>> >>>>> > > >> >  http://admintweets.com
>>>>>> >>>>> > > >> >--------------------------------------
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >>
>>>>>> >>>>> > > >
>>>>>> >>>>> > > >
>>>>>> >>>>> > > > --
>>>>>> >>>>> > > >
>>>>>> >>>>> > > > Andrija Panić
>>>>>> >>>>> > > > --------------------------------------
>>>>>> >>>>> > > >   http://admintweets.com
>>>>>> >>>>> > > > --------------------------------------
>>>>>> >>>>> > > >
>>>>>> >>>>> > >
>>>>>> >>>>> > >
>>>>>> >>>>> > >
>>>>>> >>>>> > > --
>>>>>> >>>>> > >
>>>>>> >>>>> > > Andrija Panić
>>>>>> >>>>> > > --------------------------------------
>>>>>> >>>>> > >   http://admintweets.com
>>>>>> >>>>> > > --------------------------------------
>>>>>> >>>>> > >
>>>>>> >>>>> >
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> --
>>>>>> >>>>>
>>>>>> >>>>> Andrija Panić
>>>>>> >>>>> --------------------------------------
>>>>>> >>>>>   http://admintweets.com
>>>>>> >>>>> --------------------------------------
>>>>>> >>>>>
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daan
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>
>>
>>
>> --
>> Daan
>
>
>
> --
> Daan



-- 
Daan

Mime
View raw message