cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Weller <swel...@ena.com.INVALID>
Subject Re: Cloudstack 4.9 CentOS Template failure - Connection Reset
Date Wed, 19 Jul 2017 13:41:05 GMT
Taylor,


To me this sounds like you might have some sort of networking problem, such as MTU path discovery
being broken and some device in the path setting a do not fragment flag.

Can you give us a bit more info about how you are connected to the internet as Dag has suggested?
Is there a firewall in front of your switch? Are you double NATing the traffic?


- Si




________________________________
From: Taylor <tschneider@live.com>
Sent: Wednesday, July 19, 2017 8:09 AM
To: users@cloudstack.apache.org
Cc: Taylor
Subject: RE: Cloudstack 4.9 CentOS Template failure - Connection Reset

forgot to cc myself


-------- Original message --------
From: Taylor <tschneider@live.com>
Date: 7/19/17 08:08 (GMT-06:00)
To: users@cloudstack.apache.org
Subject: RE: Cloudstack 4.9 CentOS Template failure - Connection Reset

Hey Dag,

I have tried both. The health check is good and the vm behavior does not change after recreation.

I think the problem is the network latency or the vm's resource allocation.

The download will work but a connection timeout occurs every 20MB so it needs to be done in
pieces. The hypervisors which hosts the vm is able to download without a problem.

I am on a 100mbps switch. In the past I was using a 1000mbps.

Any other thoughts on debug or work around?

I think adding retry logic should be a simple fix?


-------- Original message --------
From: Dag Sonstebo <Dag.Sonstebo@shapeblue.com>
Date: 7/19/17 03:11 (GMT-06:00)
To: users@cloudstack.apache.org
Subject: Re: Cloudstack 4.9 CentOS Template failure - Connection Reset

Hi Taylor,

This is most likely an issue with your environment rather than a bug. Take a look at your
public network and how that is connected to the internet. You have to let CloudStack pull
down the template, it’s difficult to manually populate this.

A couple of other things to try:
- recreate the SSVM – simply delete it and CloudStack will generate a new one.
- from internally in the SSVM you can also run the SSVM check script, which will do some basic
health checks for you:

root@s-2-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
56 bytes from 8.8.8.8: icmp_seq=0 ttl=53 time=24.146 ms
56 bytes from 8.8.8.8: icmp_seq=1 ttl=53 time=22.320 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 22.320/23.233/24.146/0.913 ms
Good: Can ping DNS server
================================================
Good: DNS resolves download.cloud.com
================================================
nfs is currently mounted
Mount point is /mnt/SecStorage/a833f5f1-1c6d-3e54-9a55-1fc9b7875c54
Good: Can write to mount point
================================================
Management server is 10.10.45.2. Checking connectivity.
Good: Can connect to management server port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 19/07/2017, 05:29, "Taylor" <tschneider@live.com> wrote:

    Hello,


    I am experiencing an issue while trying to download the CentOS template.


    It seems the connection is timing out and then failing.


    To debug I logged into the SSVM and tried running a wget from the nfs directory mounted
on that vm. This also failed due to connection reset.


    Wget will eventually succeed if i use retry logic as follows:


    wget http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2 --read-timeout=10


    Using this setting the download will complete in pieces after multiple timeouts (logs
below).


    Is there a work around to add retry logic? Can I manually download and add the template
to the database and restart the service? How can I file a bug report?


    Thanks,


    Taylor


    =============================================================================

    LOGS:

    =============================================================================

    root@s-103-VM:/mnt/SecStorage/5d8e791e-01cc-3d7c-84d8-f469944056e0/template/tmpl/1/5#
wget http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2 --read-timeout=10
    --2017-07-19 03:25:10--  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Resolving download.cloud.com (download.cloud.com)... 54.231.81.40
    Connecting to download.cloud.com (download.cloud.com)|54.231.81.40|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 374730926 (357M) [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    19% [============================>                                                
                                                                       ] 74,734,204  --.-K/s
  in 97s

    2017-07-19 03:26:55 (756 KB/s) - Read error at byte 74734204/374730926 (Connection timed
out). Retrying.

    --2017-07-19 03:26:56--  (try: 2)  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|54.231.81.40|:80... failed: Connection
timed out.
    Resolving download.cloud.com (download.cloud.com)... 52.216.81.16
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 299996722 (286M) remaining [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    24% [+++++++++++++++++++++++++++++======>                                         
                                                                     ] 93,094,824  --.-K/s
  in 24s

    2017-07-19 03:28:23 (740 KB/s) - Read error at byte 93094824/374730926 (Connection timed
out). Retrying.

    --2017-07-19 03:28:25--  (try: 3)  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 281636102 (269M) remaining [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    29% [++++++++++++++++++++++++++++++++++++======>                                  
                                                                     ] 112,111,876 --.-K/s
  in 26s

    2017-07-19 03:29:23 (702 KB/s) - Read error at byte 112111876/374730926 (Connection timed
out). Retrying.

    --2017-07-19 03:29:26--  (try: 4)  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 262619050 (250M) remaining [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    34% [+++++++++++++++++++++++++++++++++++++++++++=======>                          
                                                                     ] 130,790,615 --.-K/s
  in 26s

    2017-07-19 03:30:23 (712 KB/s) - Read error at byte 130790615/374730926 (Connection timed
out). Retrying.

    --2017-07-19 03:30:27--  (try: 5)  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 243940311 (233M) remaining [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    49% [+++++++++++++++++++++++++++++++++++++++++++++++++++====================>     
                                                                     ] 185,802,362 --.-K/s
  in 49s

    2017-07-19 03:31:23 (1.08 MB/s) - Read error at byte 185802362/374730926 (Connection timed
out). Retrying.

    --2017-07-19 03:31:28--  (try: 6)  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.81.16|:80... failed: Connection
timed out.
    Resolving download.cloud.com (download.cloud.com)... 52.216.225.48
    Connecting to download.cloud.com (download.cloud.com)|52.216.225.48|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 188928564 (180M) remaining [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    60% [++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++================>
                                                         ] 226,947,877 --.-K/s   in 32s

    2017-07-19 03:33:11 (1.21 MB/s) - Read error at byte 226947877/374730926 (Connection timed
out). Retrying.

    --2017-07-19 03:33:17--  (try: 7)  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.225.48|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 147783049 (141M) remaining [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    65% [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++======>
                                                  ] 247,259,880 --.-K/s   in 28s

    2017-07-19 03:33:46 (697 KB/s) - Read error at byte 247259880/374730926 (Connection timed
out). Retrying.

    --2017-07-19 03:33:53--  (try: 8)  http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2
    Connecting to download.cloud.com (download.cloud.com)|52.216.225.48|:80... connected.
    HTTP request sent, awaiting response... 206 Partial Content
    Length: 374730926 (357M), 127471046 (122M) remaining [binary/octet-stream]
    Saving to: `centos56-x86_64.vhd.bz2'

    100%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++==================================================>]
374,730,926 1010K/s   in 2m 10s





Dag.Sonstebo@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message