cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirk Kosinski <kirkkosin...@gmail.com>
Subject Re: intermittent packet loss after upgrading and restarting networks
Date Fri, 22 Aug 2014 04:24:12 GMT
Hi, did you check the documentation?  Specifically the Network
Throttling section [1].  In CloudStack the throttling can be configured
in a variety of places and the net effect can be affected by network
type and hypervisor, so it is hard to determine.  If you haven't
already, check out the doc should since it might be helpful.

Kirk

[1]
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Admin_Guide/network-rate.html

On 08/17/2014 04:38 PM, Nick Burke wrote:
> Another update:
> 
> 
> 100% confirmed to be traffic shapping set by CloudStack. I don't know
> where/how/why, and I'd love some help with this. Should I create a new
> thread? As previously mentioned, I don't believe I've set a cap of below
> 100Mbs ANYWHERE in Cloudstack. Not in compute offerings, network offerings,
> and not in the default throttle (which is set at 200).
> 
> What am I missing?
> 
> I removed tc rules on the host for two test instances and bandwidth shot up.
> 
> Before:
> 
> ubuntu@testserver01:~$ iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  4] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59276
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.4 sec  6.62 MBytes  5.35 Mbits/sec
> [  5] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59277
> [  5]  0.0-10.5 sec  6.62 MBytes  5.28 Mbits/sec
> [  4] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59278
> [  4]  0.0-10.4 sec  6.62 MBytes  5.37 Mbits/sec
> [  5] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59291
> [  5]  0.0-10.3 sec  6.62 MBytes  5.37 Mbits/sec
> [  4] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59306
> [  4]  0.0-10.5 sec  6.62 MBytes  5.30 Mbits/sec
> 
> Removed the rules for two instances on the same host:
> 
> ubuntu@dom02:~$ sudo tc qdisc del dev vnet1 root
> ubuntu@dom02:~$ sudo tc qdisc del dev vnet3 root
> ubuntu@dom02:~$ sudo tc qdisc del dev vnet3 ingress
> ubuntu@dom02:~$ sudo tc qdisc del dev vnet1 ingress
> ubuntu@dom02:~$ tc -s qdisc ls dev vnet1
> qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
>  Sent 7136572 bytes 1048 pkt (dropped 0, overlimits 0 requeues 0)
>  backlog 0b 0p requeues 0
> 
> And all of a sudden, those two instances are at blazing speeds:
> 
> ubuntu@testserver01:~$ iperf -s
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> [  4] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59322
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.0 sec  14.8 GBytes  12.7 Gbits/sec
> [  5] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59329
> [  5]  0.0-10.0 sec  19.1 GBytes  16.4 Gbits/sec
> [  4] local 10.1.1.101 port 5001 connected with 10.1.1.102 port 59330
> [  4]  0.0-10.0 sec  19.0 GBytes  16.3 Gbits/sec
> 
> 
> 
> 
> 
> On Sun, Aug 17, 2014 at 12:46 PM, Nick Burke <nick@nickburke.com> wrote:
> 
>> First,
>>
>> THANK YOU FOR REPLYING!
>>
>> Second, yes, it's currently set at 200.
>>
>> The compute offering for network is either blank (or when I tested it,
>> 1000)
>> The network offering for network limit is either 100, 1000, or blank.
>>
>>
>> Those are the only network throttling parameters that I'm aware of, are
>> there any others that I missed? Is it possible disk i/o is for some reason
>> coming into play here?
>>
>> This happens regardless of if the instance network is either a virtual
>> router or is directly connected to a vlan(ie, no virtual router) when two
>> instances are directly connected to each other.
>>
>>
>>
>>
>>
>> On Sun, Aug 17, 2014 at 12:09 PM, ilya musayev <
>> ilya.mailing.lists@gmail.com> wrote:
>>
>>> Nick
>>>
>>> Have you checked network throttle settings in "global setting" and where
>>> ever else it may be defined?
>>>
>>> regads
>>> ilya
>>>
>>> On 8/17/14, 11:27 AM, Nick Burke wrote:
>>>
>>>> Update:
>>>>
>>>> After running nperf on same instances on the same virtual network, it
>>>> looks
>>>> like all instances can get no more than 2Mb/s. Additionally, it's
>>>> sporadic
>>>> and ranges from <1Mb/s, but never more than 2Mb/s:
>>>>
>>>> user@localhost:~$ iperf -c 10.1.0.1 -d
>>>> ------------------------------------------------------------
>>>> Server listening on TCP port 5001
>>>> TCP window size: 85.3 KByte (default)
>>>> ------------------------------------------------------------
>>>> ------------------------------------------------------------
>>>> Client connecting to 10.1.0.1, TCP port 5001
>>>> TCP window size: 86.8 KByte (default)
>>>> ------------------------------------------------------------
>>>> [  5] local 10.1.0.10 port 50432 connected with 10.1.0.1 port 5001
>>>> [ ID] Interval       Transfer     Bandwidth
>>>> [  5]  0.0-11.0 sec  1.25 MBytes   950 Kbits/sec
>>>> [  4] local 10.1.0.10 port 5001 connected with 10.1.0.1 port 53839
>>>> [  4]  0.0-11.1 sec  2.50 MBytes  1.89 Mbits/sec
>>>> user@localhost:~$ iperf -c 10.1.0.1 -d
>>>> ------------------------------------------------------------
>>>> Server listening on TCP port 5001
>>>> TCP window size: 85.3 KByte (default)
>>>> ------------------------------------------------------------
>>>> ------------------------------------------------------------
>>>> Client connecting to 10.1.0.1, TCP port 5001
>>>> TCP window size: 50.3 KByte (default)
>>>> ------------------------------------------------------------
>>>> [  5] local 10.1.0.10 port 52248 connected with 10.1.0.1 port 5001
>>>> [ ID] Interval       Transfer     Bandwidth
>>>> [  5]  0.0-12.6 sec  1.25 MBytes   834 Kbits/sec
>>>> [  4] local 10.1.0.10 port 5001 connected with 10.1.0.1 port 53840
>>>> [  4]  0.0-11.9 sec  2.13 MBytes  1.49 Mbits/sec
>>>>
>>>>
>>>>
>>>> On Fri, Aug 15, 2014 at 11:40 AM, Nick Burke <nick@nickburke.com> wrote:
>>>>
>>>>  I upgraded from 4.0 to 4.3.0 some time ago. I didn't restart anything
>>>>> and
>>>>> it was all working great. However, I had to perform some maintenance
and
>>>>> had to restart everything. Now, I'm seeing packet loss on all virtuals,
>>>>> even ones on the same host.
>>>>>
>>>>> sudo ping -c 500  -f 172.20.1.1
>>>>> PING 172.20.1.1 (172.20.1.1) 56(84) bytes of data.
>>>>> ........................................
>>>>> --- 172.20.1.1 ping statistics ---
>>>>> 500 packets transmitted, 460 received, 8% packet loss, time 864ms
>>>>> rtt min/avg/max/mdev = 0.069/0.218/1.290/0.139 ms, ipg/ewma 1.731/0.328
>>>>> ms
>>>>>
>>>>> No interface errors reported anywhere. The host itself isn't under load
>>>>> at
>>>>> all. Doesn't matter if the instance uses e1000 or virtio for the
>>>>> drivers.
>>>>> The only thing that I'm aware of that changed was that I had to reboot
>>>>> all
>>>>> the physical servers.
>>>>>
>>>>>
>>>>> Could be related, but I was hit with the
>>>>>
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-6464
>>>>>
>>>>> bug. I did follow with Marcus' suggestion:
>>>>>
>>>>>
>>>>> *"This is a shot in the dark, but there have been some issues around
>>>>>
>>>>> upgrades that involve the cloud.vlan table expected contents changing.
>>>>> New
>>>>> 4.3 installs using vlan isolation don't seem to reproduce the issue.
>>>>> I'll
>>>>> see if I can reproduce anything like this with basic and/or non-vlan
>>>>> isolated upgrades/installs. Can anyone experiencing an issue look at
>>>>> their
>>>>> database via something like "select * from cloud.vlan" and look at the
>>>>> vlan_id. If you see something like "untagged" instead of
>>>>> "vlan://untagged",
>>>>> please try changing it and see if that helps."*
>>>>>
>>>>> --
>>>>> Nick
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *'What is a human being, then?' 'A seed' 'A... seed?' 'An acorn that
is
>>>>>
>>>>> unafraid to destroy itself in growing into a tree.' -David Zindell, A
>>>>> Requiem for Homo Sapiens*
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Nick
>>
>>
>>
>>
>>
>> *'What is a human being, then?' 'A seed' 'A... seed?' 'An acorn that is
>> unafraid to destroy itself in growing into a tree.' -David Zindell, A
>> Requiem for Homo Sapiens*
>>
> 
> 
> 

Mime
View raw message