Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of lusis.org@gmail.com designates
 209.85.212.173 as permitted sender)
MIME-Version: 1.0
Reply-To: cloudstack-users@lusis.org
Sender: lusis.org@gmail.com
In-Reply-To: 
 <CAPy8ofK4+0WnAakoZAr80pBU9yU5oAX4x2F0CJA4WGc=XZ8AAg@mail.gmail.com>
References: 
 <CAPy8ofK4+0WnAakoZAr80pBU9yU5oAX4x2F0CJA4WGc=XZ8AAg@mail.gmail.com>
Date: Tue, 7 Jan 2014 21:41:45 -0500
Message-ID: 
 <CAPy8of+Nqqc_90taUPuZrdSbjB-XVvYd2b=-JY+ACXFAXAG1LA@mail.gmail.com>
Subject: Re: unable to delete shared network
From: John Vincent <cloudstack-users@lusis.org>
To: users@cloudstack.apache.org
Content-Type: multipart/alternative; boundary=047d7bfcf35c0d55d704ef6c6fae

--047d7bfcf35c0d55d704ef6c6fae
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I wanted to follow up on this a bit. I've been able to duplicate this
multiple times now (and resolve the issue myself).

First the resolution: setting the "removed" timestamp in the nics table for
the placeholder entry allows me to delete the network. My suspicion without
looking at the code is that the check for entries to cleanup is partially
filtering by results without a "removed" timestamp. I haven't verified this
in the code.

As for duplicating it, when I created a shared network the vrouter isn't
immediately started until I either launch into it or "restart network".
Sane behaviour and I recall this being tunable. Regardless, I'm seeing
weird vrouter provisioning issues where it XenServer isn't apparently
getting the message to start the instance. Eventually the vrouter is left
in a stopped state. If I start it then it starts immediately. This MAY be
something I need to troubleshoot more on my side but I want to put it aside
for a moment.

Once I start the vrouter (and I can launch instances fine into that shared
network and get out), I still cannot delete the network regardless of any
instances in it or vrouter being destroyed or not. If there are no
instances and I attempt to delete the network, the vrouter is being
terminated fine but that last step I documented previously is still
happening until I make the aforementioned change in the nics table.


On Tue, Jan 7, 2014 at 12:22 PM, John Vincent <cloudstack-users@lusis.org>w=
rote:

> I brought this issue up on irc but figured it was a good idea to bring it
> up here as well. I'm running into an issue deleting a shared network. Thi=
s
> is on CS 4.2.0. I have two shared networks exibiting the same behavior. A=
t
> this point I'd like to clean them up in the database if possible but if
> something is off with the nics table (as I suspect), I'd be willing to fi=
x
> that as well.
>
> From the logs it appears that the NPE is in the cleanup step:
>
> 2014-01-07 04:48:15,111 DEBUG [network.lb.LoadBalancingRulesManagerImpl] =
(Job-Executor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Foun=
d 0 lb rules to cleanup
> 2014-01-07 04:48:15,111 DEBUG [cloud.network.NetworkManagerImpl] (Job-Exe=
cutor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Cleaning up =
remote access vpns as a part of
>  public IP id=3D4 release...
> 2014-01-07 04:48:15,122 DEBUG [network.vpn.RemoteAccessVpnManagerImpl] (J=
ob-Executor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) there =
are no Remote access vpns for p
> ublic ip address id=3D4
> 2014-01-07 04:48:15,132 DEBUG [cloud.network.NetworkManagerImpl] (Job-Exe=
cutor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Sending dest=
roy to com.cloud.network.elemen
> t.VirtualRouterElement_EnhancerByCloudStack_c958fdcb@31e924f9
> 2014-01-07 04:48:15,133 DEBUG [cloud.network.NetworkManagerImpl] (Job-Exe=
cutor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Network id=
=3D207 is destroyed successfully, c
> leaning up corresponding resources now.
> 2014-01-07 04:48:15,138 DEBUG [network.guru.DirectNetworkGuru] (Job-Execu=
tor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Releasing ip 1=
72.16.0.1 of placeholder nic Ni
> c[54-null-null-172.16.0.1]
> 2014-01-07 04:48:15,140 DEBUG [db.Transaction.Transaction] (Job-Executor-=
26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Rolling back the t=
ransaction: Time =3D 5 Name =3D  -A
> syncJobManagerImpl$1.run:494-Executors$RunnableAdapter.call:471-FutureTas=
k$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:1145-Th=
readPoolExecutor$Worker.run:6
> 15-Thread.run:724; called by -Transaction.rollback:898-Transaction.remove=
UpTo:841-Transaction.close:665-TransactionContextBuilder.interceptException=
:63-ComponentInstantiationPos
> tProcessor$InterceptorDispatcher.intercept:133-NetworkManagerImpl.destroy=
Network:3131-ComponentInstantiationPostProcessor$InterceptorDispatcher.inte=
rcept:125-NetworkServiceImpl.
> deleteNetwork:1767-ComponentInstantiationPostProcessor$InterceptorDispatc=
her.intercept:125-DeleteNetworkCmd.execute:70-ApiDispatcher.dispatch:158-As=
yncJobManagerImpl$1.run:531
> 2014-01-07 04:48:15,146 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Exec=
utor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Unexpected
> exception while executing org.apache.cloudstack.api.command.user.network.=
DeleteNetworkCmd
> java.lang.NullPointerException
>         at com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGu=
ru.java:311)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$=
InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:12=
5)
>         at com.cloud.network.NetworkManagerImpl.destroyNetwork(NetworkMan=
agerImpl.java:3131)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$=
InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:12=
5)
>         at com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServ=
iceImpl.java:1767)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$=
InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:12=
5)
>         at org.apache.cloudstack.api.command.user.network.DeleteNetworkCm=
d.execute(DeleteNetworkCmd.java:70)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
>         at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.=
java:531)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.=
java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:=
334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx=
ecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
> 2014-01-07 04:48:15,149 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Exec=
utor-26:job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ]) Complete asyn=
c job-320 =3D [ ceb8c1fb-cb28-42b6-b9f6-6d82f760a689 ], jobStatus: 2, resul=
tCode: 530, result: Error Code: 530 Error text: null
>
>
> I've tried this with and without the virtual router running for that netw=
ork. Here are the entries from the nic table for that network right now (wi=
th the VR running):
>
>
> mysql> select id,instance_id,reservation_id,ip4_address,state,strategy fr=
om nics where network_id =3D '208';
> +-----+-------------+----------------+--------------+--------------+-----=
--------+
> | id  | instance_id | reservation_id | ip4_address  | state        | stra=
tegy    |
> +-----+-------------+----------------+--------------+--------------+-----=
--------+
> |  58 |          26 | NULL           | 172.16.1.79  | Deallocating | Crea=
te      |
> |  60 |        NULL | NULL           | 172.16.1.1   | Reserved     | Plac=
eHolder |
> |  61 |          27 | NULL           | 172.16.1.1   | Deallocating | Crea=
te      |
> |  64 |          28 | NULL           | 172.16.1.45  | Deallocating | Crea=
te      |
> |  73 |          31 | NULL           | 172.16.1.141 | Deallocating | Crea=
te      |
> |  77 |          33 | NULL           | 172.16.1.1   | Deallocating | Crea=
te      |
> | 147 |          85 | NULL           | 172.16.1.1   | Deallocating | Crea=
te      |
> | 150 |          86 | NULL           | 172.16.1.1   | Reserved     | Crea=
te      |
> +-----+-------------+----------------+--------------+--------------+-----=
--------+
> 8 rows in set (0.00 sec)
>
>
> Here's the entry from the networks table for it:
>
>
> mysql> select * from networks where id =3D 208\G
> *************************** 1. row ***************************
>                    id: 208
>                  name: prod-be-network
>                  uuid: be81f804-f75d-4c8f-89af-9800b3c3f328
>          display_text: Production Backend Network
>          traffic_type: Guest
> broadcast_domain_type: Vlan
>         broadcast_uri: vlan://1121
>               gateway: 172.16.1.1
>                  cidr: 172.16.1.0/24
>                  mode: Dhcp
>   network_offering_id: 19
>   physical_network_id: 202
>        data_center_id: 1
>             guru_name: DirectNetworkGuru
>                 state: Implementing
>               related: 208
>             domain_id: 1
>            account_id: 1
>                  dns1: 8.8.8.8
>                  dns2: NULL
>             guru_data: NULL
>            set_fields: 0
>              acl_type: Domain
>        network_domain: cs1cloud.internal
>        reservation_id: eed95258-e478-49ed-a5ef-ffa5460d320d
>            guest_type: Shared
>      restart_required: 0
>               created: 2013-11-20 16:00:26
>               removed: NULL
>     specify_ip_ranges: 1
>                vpc_id: NULL
>           ip6_gateway: NULL
>              ip6_cidr: NULL
>          network_cidr: NULL
>       display_network: 1
>        network_acl_id: NULL
> 1 row in set (0.00 sec)
>
>
> For comparison, here's the entry from nics and networks for the other net=
work that won't delete (207)
>
>
> nics:
>
> mysql> select id,instance_id,reservation_id,ip4_address,state,strategy fr=
om nics where network_id =3D '207';
> +-----+-------------+----------------+--------------+--------------+-----=
--------+
> | id  | instance_id | reservation_id | ip4_address  | state        | stra=
tegy    |
> +-----+-------------+----------------+--------------+--------------+-----=
--------+
> |  53 |          24 | NULL           | 172.16.0.23  | Deallocating | Crea=
te      |
> |  54 |        NULL | NULL           | 172.16.0.1   | Reserved     | Plac=
eHolder |
> |  55 |          25 | NULL           | 172.16.0.1   | Deallocating | Crea=
te      |
> |  59 |          26 | NULL           | 172.16.0.80  | Deallocating | Crea=
te      |
> |  72 |          31 | NULL           | 172.16.0.164 | Deallocating | Crea=
te      |
> |  74 |          32 | NULL           | 172.16.0.1   | Deallocating | Crea=
te      |
> |  80 |          34 | NULL           | 172.16.0.36  | Deallocating | Crea=
te      |
> |  81 |          35 | NULL           | 172.16.0.32  | Deallocating | Crea=
te      |
> |  85 |          38 | NULL           | 172.16.0.168 | Deallocating | Crea=
te      |
> | 144 |          84 | NULL           | 172.16.0.1   | Deallocating | Crea=
te      |
> +-----+-------------+----------------+--------------+--------------+-----=
--------+
> 10 rows in set (0.00 sec)
>
>
> networks:
>
> mysql> select * from networks where id =3D 207\G
> *************************** 1. row ***************************
>                    id: 207
>                  name: prod-fe-network
>                  uuid: da81b864-7082-4c46-a425-c5c6265a1eca
>          display_text: Production frontend network
>          traffic_type: Guest
> broadcast_domain_type: Vlan
>         broadcast_uri: vlan://1120
>               gateway: 172.16.0.1
>                  cidr: 172.16.0.0/24
>                  mode: Dhcp
>   network_offering_id: 19
>   physical_network_id: 202
>        data_center_id: 1
>             guru_name: DirectNetworkGuru
>                 state: Setup
>               related: 207
>             domain_id: 1
>            account_id: 1
>                  dns1: 8.8.8.8
>                  dns2: NULL
>             guru_data: NULL
>            set_fields: 0
>              acl_type: Domain
>        network_domain: cs1cloud.internal
>        reservation_id: 81d652a1-9cb3-4833-92ca-58ee7b33c04b
>            guest_type: Shared
>      restart_required: 0
>               created: 2013-11-20 15:54:39
>               removed: NULL
>     specify_ip_ranges: 1
>                vpc_id: NULL
>           ip6_gateway: NULL
>              ip6_cidr: NULL
>          network_cidr: NULL
>       display_network: 1
>        network_acl_id: NULL
> 1 row in set (0.00 sec)
>
>

--047d7bfcf35c0d55d704ef6c6fae--