Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
From: Simon Weller <sweller@ena.com>
To: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Subject: Re: Router VM: patchviasocket.py timeout issue on 1 out of 4 networks
Thread-Topic: Router VM: patchviasocket.py timeout issue on 1 out of 4
 networks
Thread-Index: AQHSVOe2trYkke1ytEG2Yh4adlKuzqEFJ3It
Date: Tue, 13 Dec 2016 02:31:41 +0000
Message-ID: <BY2PR02MB2007CD95B7F940BA9A1214AFA99B0@BY2PR02MB2007.namprd02.prod.outlook.com>
References: <CAOJ8eeCD8xpLR+wsh1_esJ5ksLywwSU8qjpT_qRbeT-S-F0xuQ@mail.gmail.com>
In-Reply-To: <CAOJ8eeCD8xpLR+wsh1_esJ5ksLywwSU8qjpT_qRbeT-S-F0xuQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [132.245.243.37]
x-ms-office365-filtering-correlation-id: 67017005-5d21-4b4b-c914-08d423002c18
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001);SRVR:BY2PR02MB2008;
x-microsoft-exchange-diagnostics: 1;BY2PR02MB2008;7:uZw3l0HKysal6LT0NR34ZcUJgccizdD4R4WoVtcNKS0RaBMEwPcvJ0Pnl9+ZNxqUiSJuDd3yQJg+PYJZvHw4GqZs/i1KZnPEgt1HZrr718XOrHyMGdB+8DAgtWsUKIEEMPdSSbsFkJZkhXu7u5hncE6QV9N4l0oX4T7/ObQgrEp96CLJ0EdSEPKotqpLK88ciW/ieQpM02bt+fNPQGP8KowimOwlog/V3WhUUc4msb7hy7JIuBMw0ed9aJkATHPF6apnXBtGy3ev6mBrDMDs1wN8QVp1GvrgC3u9cF4ps6Sbpl0JJ3SlBJqhwQN+GvghNXo3TsaVrSaCw42xfY4fIw4/vohm05WzrtGZfEa6s80mosdyLBNhn2JCh+w2Zi1LvqMjKFzu+F45he1NyIeVKgqgKGnA13txtlxAiCzW7xAQL6B0crv6/YOK0yN9JBAAJrkpgWQLdt/uhn33kL+EKg==
x-microsoft-antispam-prvs: <BY2PR02MB200809FD2FAD11BE6A0C660AA99B0@BY2PR02MB2008.namprd02.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:;
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123562025)(20161123564025)(20161123560025)(20161123555025)(6072148);SRVR:BY2PR02MB2008;BCL:0;PCL:0;RULEID:;SRVR:BY2PR02MB2008;
x-forefront-prvs: 01559F388D
x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(6009001)(7916002)(39850400002)(39410400002)(39840400002)(39450400003)(377424004)(377454003)(189002)(199003)(74316002)(86362001)(7696004)(575784001)(2906002)(76576001)(7736002)(76176999)(1730700003)(122556002)(6116002)(50986999)(68736007)(92566002)(2900100001)(77096006)(102836003)(33656002)(38730400001)(54356999)(229853002)(101416001)(3846002)(6506006)(19627405001)(97736004)(107886002)(81166006)(8676002)(3280700002)(3660700001)(4001150100001)(6436002)(81156014)(66066001)(3900700001)(8936002)(106116001)(6606003)(106356001)(99286002)(2351001)(189998001)(2950100002)(2501003)(105586002)(110136003)(6916009)(9686002)(5660300001)(270524002);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR02MB2008;H:BY2PR02MB2007.namprd02.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en;
received-spf: None (protection.outlook.com: ena.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_BY2PR02MB2007CD95B7F940BA9A1214AFA99B0BY2PR02MB2007namp_"
MIME-Version: 1.0
X-OriginatorOrg: ena.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Dec 2016 02:31:41.1105
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 6dc38cd4-4d4f-4826-9649-17854289d170
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR02MB2008
X-ENA-MailScanner-Information: Please contact support@ena.com for more information
X-ENA-MailScanner-ID: D641B1480A53.AF0B0
X-ENA-MailScanner: No viruses found
X-ENA-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-2.5,
	required 4, BAYES_00 -3.20, DKIM_SIGNED 0.10, DKIM_VALID -0.10,
	DKIM_VALID_AU -0.20, HTML_MESSAGE 1.20, OS_UNKNOWN -0.10,
	SPF_HELO_PASS -0.20)
X-ENA-MailScanner-From: sweller@ena.com
X-ENA-MailScanner-Watermark: 1482201105.64673@SiXSMgNe+5aMnB+qZTJHfw
archived-at: Tue, 13 Dec 2016 02:32:21 -0000

--_000_BY2PR02MB2007CD95B7F940BA9A1214AFA99B0BY2PR02MB2007namp_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Can you turn on agent debug mode and take a look at the debug level logs?


You can do that by running sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/lo=
g4j-cloud.xml on the host and then restarting the agent.


- Si


________________________________
From: Syahrul Sazli Shaharir <sazli@pulasan.my>
Sent: Monday, December 12, 2016 8:21 PM
To: users@cloudstack.apache.org
Subject: Router VM: patchviasocket.py timeout issue on 1 out of 4 networks

Hi,

I am running latest Cloudstack 4.9.0.1 on CentOS 7 KVM + ceph
environment. After running for some time, I faced with an issue with
one out of 4 networks - following a heartbeat-induced reset on all
hosts, the associated virtual router would not get recreated and
started properly on any of the 3 hosts I have, even after repeated
attempts of the following:-
- destroy-recreate cycles, via Cloudstack UI
- restartNetwork cleanup=3Dtrue API calls (failed with errorcode =3D 530).
- redownload and reregister system VM template as another entry and
assign to router VM in global setting (boots the new template OK, but
still same problem)
- tweak default system offering for router VM (increased RAM from 256 to 51=
2MB)
- created new system offering, with RAM tweak, and use of ceph rbd
store, and assigned it to Cloud.Com-SoftwareRouter as per docs - which
didnt work for some reason: it kept on using initial default offering
and created image on local host storage
- upgrade to latest cloudstack (previously was running 4.8)

As with a handful of others in this list archives, virsh list and
dumpxml shows the VM created OK but failed soon after booting, as
found in the following error in agent.log :-

2016-12-13 10:03:33,894 WARN  [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) (logid:633e6e03) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-668-VM -p %template=3DdomP%name=3Dr-668-VM%eth0ip=3D10.3.28.10%eth0mas=
k=3D255.255.255.0%gateway=3D10.3.28.1%domain=3Dnocser.net%cidrsize=3D24%dhc=
prange=3D10.3.28.1%eth1ip=3D169.254.0.33%eth1mask=3D255.255.0.0%type=3Ddhcp=
srvr%disable_rp_filter=3Dtrue%dns1=3D8.8.8.8%dns2=3D8.8.4.4%ip6dns1=3D%ip6d=
ns2=3D%baremetalnotificationsecuritykey=3DuavJByNGGjNLrELG-qbdN99__1I3tnp8q=
a0KbcsKokKJcPB43K9s6oQu2nMLqo3YP8p6jqDy5XT3WWOWBA2yNw%baremetalnotification=
apikey=3D8JH4mdkxsEMhgIBgMonkNXAEKjVOeZnG1m5UVekvvo4v_iXQ4ZS7rh6NNS0qphhc7Z=
rCauiz23tp2-Wa3AASlg%host=3D10.2.30.11%port=3D8080
.  Output is:
.....
2016-12-13 10:05:45,895 WARN  [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) (logid:633e6e03) Timed out:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
vr_cfg.sh 169.254.0.33 -c
/var/cache/cloud/VR-48ea8a95-6c02-499f-88d3-eae5bf9f9fbe.cfg .  Output
is:

As mentioned, this only happens with 1 network (always the same
network). The other router VMs work OK. Any clues on how to
troubleshoot this further, would be greatly appreciated.

Thanks.

--
--sazli
Syahrul Sazli Shaharir <sazli@pulasan.my>

--_000_BY2PR02MB2007CD95B7F940BA9A1214AFA99B0BY2PR02MB2007namp_--