Mailing-List: contact issues-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Date: Tue, 22 Apr 2014 17:32:14 +0000 (UTC)
From: "Logan B (JIRA)" <jira@apache.org>
To: cloudstack-issues@incubator.apache.org
Message-ID: <JIRA.12709819.1398187866655.156433.1398187934697@arcas>
In-Reply-To: <JIRA.12709819.1398187866655@arcas>
References: <JIRA.12709819.1398187866655@arcas>
Subject: [jira] [Created] (CLOUDSTACK-6473) Debian 7 Virtual Router
 ip_conntrack_max not set at boot
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Logan B created CLOUDSTACK-6473:
-----------------------------------

             Summary: Debian 7 Virtual Router ip_conntrack_max not set at boot
                 Key: CLOUDSTACK-6473
                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6473
             Project: CloudStack
          Issue Type: Bug
      Security Level: Public (Anyone can view this level - this is the default.)
          Components: Virtual Router
    Affects Versions: 4.3.0
         Environment: XenServer 6.2
CloudStack 4.3.0
Debian 7 SystemVM/Virtual Router
            Reporter: Logan B
             Fix For: 4.3.1


The Problem:
The Debian 7 Virtual Router VMs for XenServer experiences intermittent connectivity problems.  This affects all VMs behind the virtual router in various ways: SSH failures, Apache connections fail, etc.

This issue also affects various function within CloudStack that attempt to connect to the Virtual Router (updating firewall rules, NAT, etc.)

The Cause:
It appears that the issues is being caused by a low default limit for the net.ipv4.netfilter.ip_conntrack_max sysctl.  The issue can be easily diagnosed in /var/log/messages:
Apr 22 15:45:34 r-5602-VM kernel: [ 1085.117498] nf_conntrack: table full, dropping packet.
Apr 22 15:45:34 r-5602-VM kernel: [ 1085.133095] nf_conntrack: table full, dropping packet.
Apr 22 15:45:34 r-5602-VM kernel: [ 1085.145440] nf_conntrack: table full, dropping packet.

The default setting for ip_conntrack_max is '3796': 
# sysctl net.ipv4.netfilter.ip_conntrack_max
net.ipv4.netfilter.ip_conntrack_max = 3796

As per /etc/sysctl.conf this setting should be '1000000':
net.ipv4.netfilter.ip_conntrack_max=1000000

It would appear that this setting is not being correctly applied when the virtual router boots.

The Solution:
- A temporary workaround is to manually set the ip_conntrack_max sysctl to the correct value:
# sysctl -w net.ipv4.netfilter.ip_conntrack_max=1000000

It's likely that this sysctl is being run at boot before the module is loaded, so it doesn't take effect.  There are various solutions suggested around the web, any of which should work fine.

To resolve this problem a new System VM template should be created.  I'm assuming this can be done in between CloudStack releases.  I know there is supposed to be a new template released to fix the HeartBleed vulnerability, so this would be a good fix to include with that updated template.


--
This message was sent by Atlassian JIRA
(v6.2#6252)