Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
From: Simon Weller <sweller@ena.com.INVALID>
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: Re: Cloudstack 4.9 CentOS Template failure - Connection Reset
Thread-Topic: Cloudstack 4.9 CentOS Template failure - Connection Reset
Thread-Index: AQHTAJAUtz5l9m2esUKFeZuepGSlP6JbH1sZgAAHwiOAAASrqYAAAovC
Date: Wed, 19 Jul 2017 14:04:29 +0000
Message-ID: <BN6PR02MB333190C4203EED26D368FA19A9A60@BN6PR02MB3331.namprd02.prod.outlook.com>
References: <DM2PR06MB3345FEFBDBC8AE4AA1BE403BEA60@DM2PR06MB334.namprd06.prod.outlook.com>,<DM2PR06MB334262A36AE1C0D0A22FFDCBEA60@DM2PR06MB334.namprd06.prod.outlook.com>,<BN6PR02MB3331285640DF1A5089587835A9A60@BN6PR02MB3331.namprd02.prod.outlook.com>,<DM2PR06MB3341B62861BF8B33155B2B5BEA60@DM2PR06MB334.namprd06.prod.outlook.com>
In-Reply-To: <DM2PR06MB3341B62861BF8B33155B2B5BEA60@DM2PR06MB334.namprd06.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_BN6PR02MB333190C4203EED26D368FA19A9A60BN6PR02MB3331namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Jul 2017 14:04:29.4635
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 6dc38cd4-4d4f-4826-9649-17854289d170
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR02MB3329
X-ENA-MailScanner-Information: Please contact support@ena.com for more information
X-ENA-MailScanner-ID: A7B091480B53.A9AF7
X-ENA-MailScanner: No viruses found
X-ENA-MailScanner-SpamCheck: not spam, SpamAssassin (not cached,
	score=-2.099, required 4, autolearn=not spam, BAYES_00 -1.90,
	DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10,
	DKIM_VALID_EF -0.10, HTML_MESSAGE 0.00, OS_UNKNOWN 0.00,
	SPF_HELO_PASS -0.00)
X-ENA-MailScanner-From: sweller@ena.com
X-ENA-MailScanner-Watermark: 1501077872.27695@zNb7W+8nvBNsBkXGinT64Q
archived-at: Wed, 19 Jul 2017 14:04:51 -0000

--_000_BN6PR02MB333190C4203EED26D368FA19A9A60BN6PR02MB3331namp_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

So if your routers are blocking all ICMP, they will break MTU path discover=
y. See this: https://en.wikipedia.org/wiki/Path_MTU_Discovery


________________________________
From: Taylor <tschneider@live.com>
Sent: Wednesday, July 19, 2017 8:57 AM
To: users@cloudstack.apache.org
Cc: Taylor
Subject: Re: Cloudstack 4.9 CentOS Template failure - Connection Reset


Hi Simon,

I a not sure about the networking problem you mentioned. I will google that=
 and if you have any quick ways to check let me know.

As for the double NAT, the answer is yes. My network is configured as follo=
ws:

Internet -- 1000MBps router -- 100Mbps router -- Hypervisor / Cloudstack / =
NFS

As far as I am aware the routers are acting as firewalls for incoming traff=
ic (they are simple home routers) but should not impact outgoing traffic.

Thanks,

Taylor

________________________________
From: Simon Weller <sweller@ena.com.INVALID>
Sent: Wednesday, July 19, 2017 9:41 AM
To: users@cloudstack.apache.org
Subject: Re: Cloudstack 4.9 CentOS Template failure - Connection Reset

Taylor,


To me this sounds like you might have some sort of networking problem, such=
 as MTU path discovery being broken and some device in the path setting a d=
o not fragment flag.

Can you give us a bit more info about how you are connected to the internet=
 as Dag has suggested? Is there a firewall in front of your switch? Are you=
 double NATing the traffic?


- Si


________________________________
From: Taylor <tschneider@live.com>
Sent: Wednesday, July 19, 2017 8:09 AM
To: users@cloudstack.apache.org
Cc: Taylor
Subject: RE: Cloudstack 4.9 CentOS Template failure - Connection Reset

forgot to cc myself


-------- Original message --------
From: Taylor <tschneider@live.com>
Date: 7/19/17 08:08 (GMT-06:00)
To: users@cloudstack.apache.org
Subject: RE: Cloudstack 4.9 CentOS Template failure - Connection Reset

Hey Dag,

I have tried both. The health check is good and the vm behavior does not ch=
ange after recreation.

I think the problem is the network latency or the vm's resource allocation.

The download will work but a connection timeout occurs every 20MB so it nee=
ds to be done in pieces. The hypervisors which hosts the vm is able to down=
load without a problem.

I am on a 100mbps switch. In the past I was using a 1000mbps.

Any other thoughts on debug or work around?

I think adding retry logic should be a simple fix?


-------- Original message --------
From: Dag Sonstebo <Dag.Sonstebo@shapeblue.com>
Date: 7/19/17 03:11 (GMT-06:00)
To: users@cloudstack.apache.org
Subject: Re: Cloudstack 4.9 CentOS Template failure - Connection Reset

Hi Taylor,

This is most likely an issue with your environment rather than a bug. Take =
a look at your public network and how that is connected to the internet. Yo=
u have to let CloudStack pull down the template, it=92s difficult to manual=
ly populate this.

A couple of other things to try:
- recreate the SSVM =96 simply delete it and CloudStack will generate a new=
 one.
- from internally in the SSVM you can also run the SSVM check script, which=
 will do some basic health checks for you:

root@s-2-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
56 bytes from 8.8.8.8: icmp_seq=3D0 ttl=3D53 time=3D24.146 ms
56 bytes from 8.8.8.8: icmp_seq=3D1 ttl=3D53 time=3D22.320 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev =3D 22.320/23.233/24.146/0.913 ms
Good: Can ping DNS server
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Good: DNS resolves download.cloud.com
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
nfs is currently mounted
Mount point is /mnt/SecStorage/a833f5f1-1c6d-3e54-9a55-1fc9b7875c54
Good: Can write to mount point
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Management server is 10.10.45.2. Checking connectivity.
Good: Can connect to management server port 8250
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Good: Java process is running
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Tests Complete. Look for ERROR or WARNING above.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 19/07/2017, 05:29, "Taylor" <tschneider@live.com> wrote:

    Hello,


    I am experiencing an issue while trying to download the CentOS template=
.


    It seems the connection is timing out and then failing.


    To debug I logged into the SSVM and tried running a wget from the nfs d=
irectory mounted on that vm. This also failed due to connection reset.


    Wget will eventually succeed if i use retry logic as follows:


    wget http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz=
2 --read-timeout=3D10


    Using this setting the download will complete in pieces after multiple =
timeouts (logs below).


    Is there a work around to add retry logic? Can I manually download and =
add the template to the database and restart the service? How can I file a =
bug report?


    Thanks,


    Taylor


    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D

    LOGS:

    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D

    root@s-103-VM:/mnt/SecStorage/5d8e791e-01cc-3d7c-84d8-f469944056e0/temp=
late/tmpl/1/5# wget http://download.cloud.com/templates/builtin/centos56-x8=
6_64.vhd.bz2 --read-timeout=3D10
    --2017-07-19 03:25:10--  http://download.cloud.com/templates/builtin/ce=
ntos56-x86_64.vhd.bz2
    Resolving download.cloud.com (download.cloud.com)... 54.231.81.40
    Connecting to download.cloud.com (download.cloud.com)|54.231.81.40|:80.=
.. connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 374730926 (357M) [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    19% [=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D>                                                        =
                                                                ] 74,734,20=
4  --.-K/s   in 97s

    2017-07-19 03:26:55 (756 KB/s) - Read error at byte 74734204/374730926 =
(Connection timed out). Retrying.

    --2017-07-19 03:26:56--  (try: 2)  http://download.cloud.com/templates/=
builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|54.231.81.40|:80.=
.. failed: Connection timed out.
    Resolving download.cloud.com (download.cloud.com)... 52.216.81.16
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80.=
.. connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 299996722 (286M) remaining [binary/octet-stre=
am]
    Saving to: `centos56-x86_64.vhd.bz2'

    24% [+++++++++++++++++++++++++++++=3D=3D=3D=3D=3D=3D>                  =
                                                                           =
                  ] 93,094,824  --.-K/s   in 24s

    2017-07-19 03:28:23 (740 KB/s) - Read error at byte 93094824/374730926 =
(Connection timed out). Retrying.

    --2017-07-19 03:28:25--  (try: 3)  http://download.cloud.com/templates/=
builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80.=
.. connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 281636102 (269M) remaining [binary/octet-stre=
am]
    Saving to: `centos56-x86_64.vhd.bz2'

    29% [++++++++++++++++++++++++++++++++++++=3D=3D=3D=3D=3D=3D>           =
                                                                           =
                  ] 112,111,876 --.-K/s   in 26s

    2017-07-19 03:29:23 (702 KB/s) - Read error at byte 112111876/374730926=
 (Connection timed out). Retrying.

    --2017-07-19 03:29:26--  (try: 4)  http://download.cloud.com/templates/=
builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80.=
.. connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 262619050 (250M) remaining [binary/octet-stre=
am]
    Saving to: `centos56-x86_64.vhd.bz2'

    34% [+++++++++++++++++++++++++++++++++++++++++++=3D=3D=3D=3D=3D=3D=3D> =
                                                                           =
                    ] 130,790,615 --.-K/s   in 26s

    2017-07-19 03:30:23 (712 KB/s) - Read error at byte 130790615/374730926=
 (Connection timed out). Retrying.

    --2017-07-19 03:30:27--  (try: 5)  http://download.cloud.com/templates/=
builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80.=
.. connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 243940311 (233M) remaining [binary/octet-stre=
am]
    Saving to: `centos56-x86_64.vhd.bz2'

    49% [+++++++++++++++++++++++++++++++++++++++++++++++++++=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>                             =
                                              ] 185,802,362 --.-K/s   in 49=
s

    2017-07-19 03:31:23 (1.08 MB/s) - Read error at byte 185802362/37473092=
6 (Connection timed out). Retrying.

    --2017-07-19 03:31:28--  (try: 6)  http://download.cloud.com/templates/=
builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80.=
.. failed: Connection timed out.
    Resolving download.cloud.com (download.cloud.com)... 52.216.225.48
    Connecting to download.cloud.com (download.cloud.com)|52.216.225.48|:80=
... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 188928564 (180M) remaining [binary/octet-stre=
am]
    Saving to: `centos56-x86_64.vhd.bz2'

    60% [++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
++++++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>                    =
                                      ] 226,947,877 --.-K/s   in 32s

    2017-07-19 03:33:11 (1.21 MB/s) - Read error at byte 226947877/37473092=
6 (Connection timed out). Retrying.

    --2017-07-19 03:33:17--  (try: 7)  http://download.cloud.com/templates/=
builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.225.48|:80=
... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 147783049 (141M) remaining [binary/octet-stre=
am]
    Saving to: `centos56-x86_64.vhd.bz2'

    65% [++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
+++++++++++++++++++++++=3D=3D=3D=3D=3D=3D>                                 =
                  ] 247,259,880 --.-K/s   in 28s

    2017-07-19 03:33:46 (697 KB/s) - Read error at byte 247259880/374730926=
 (Connection timed out). Retrying.

    --2017-07-19 03:33:53--  (try: 8)  http://download.cloud.com/templates/=
builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.225.48|:80=
... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 127471046 (122M) remaining [binary/octet-stre=
am]
    Saving to: `centos56-x86_64.vhd.bz2'

    100%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
++++++++++++++++++++++++++++++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>] 374,730,926 1010K/s   in 2m 10s


Dag.Sonstebo@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue


--_000_BN6PR02MB333190C4203EED26D368FA19A9A60BN6PR02MB3331namp_--