From dev-return-111398-archive-asf-public=cust-asf.ponee.io@cloudstack.apache.org Wed May 16 13:25:34 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 288BB180669 for ; Wed, 16 May 2018 13:25:32 +0200 (CEST) Received: (qmail 76499 invoked by uid 500); 16 May 2018 11:25:31 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 76482 invoked by uid 99); 16 May 2018 11:25:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 May 2018 11:25:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6BD1CC00CC for ; Wed, 16 May 2018 11:25:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.998 X-Spam-Level: * X-Spam-Status: No, score=1.998 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=shapeblue.onmicrosoft.com header.b=QBy5ugVo; dkim=fail (1024-bit key) reason="fail (body has been altered)" header.d=shapeblue.onmicrosoft.com header.b=gzq37xKK Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id uEYHhYVSB7ck for ; Wed, 16 May 2018 11:25:23 +0000 (UTC) Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50113.outbound.protection.outlook.com [40.107.5.113]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 418375F2C2 for ; Wed, 16 May 2018 11:25:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shapeblue.onmicrosoft.com; s=selector1-shapeblue-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h83LbyOG6eXu4VANGufnjJsyHw6yEQDa+ws0h16O4ys=; b=QBy5ugVo+uMFpXS5EWaOmv7+LZd7EHQqQX0LLjqNsl70R7EQbe8vNM82LCW++y1982R/WYmMT4sFcCp5smDqU2UPzynLW1omZMDSmBsDq4upHXoSyHIelZ1vrxprX0YGX/YWivFXpQ/t6awIUy8nvWqkcgXF7fiB78Mw1trx6gI= Received: from HE1PR0701CA0060.eurprd07.prod.outlook.com (2603:10a6:3:9e::28) by HE1PR0702MB3691.eurprd07.prod.outlook.com (2603:10a6:7:8d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.776.4; Wed, 16 May 2018 11:25:15 +0000 Received: from DB5EUR01FT054.eop-EUR01.prod.protection.outlook.com (2a01:111:f400:7e02::202) by HE1PR0701CA0060.outlook.office365.com (2603:10a6:3:9e::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.776.4 via Frontend Transport; Wed, 16 May 2018 11:25:15 +0000 Authentication-Results: spf=fail (sender IP is 104.40.179.195) smtp.mailfrom=shapeblue.com; cloudstack.apache.org; dkim=fail (body hash did not verify) header.d=shapeblue.onmicrosoft.com;cloudstack.apache.org; dmarc=none action=none header.from=shapeblue.com; Received-SPF: Fail (protection.outlook.com: domain of shapeblue.com does not designate 104.40.179.195 as permitted sender) receiver=protection.outlook.com; client-ip=104.40.179.195; helo=smtpworker-in-13.xware-eu-1.o365.crossware.co.nz; Received: from smtpworker-in-13.xware-eu-1.o365.crossware.co.nz (104.40.179.195) by DB5EUR01FT054.mail.protection.outlook.com (10.152.5.133) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.20.755.15 via Frontend Transport; Wed, 16 May 2018 11:25:13 +0000 Received: from EUR01-HE1-obe.outbound.protection.outlook.com (213.199.154.214) by smtpworker-in-13.xware-eu-1.o365.crossware.co.nz with Crossware for Office365; Wed, 16 May 2018 11:25:13 +0000 Received: from AM4PR07MB3490.eurprd07.prod.outlook.com (10.171.190.27) by AM4PR07MB3172.eurprd07.prod.outlook.com (10.171.188.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.776.4; Wed, 16 May 2018 11:25:10 +0000 Received: from AM4PR07MB3490.eurprd07.prod.outlook.com ([fe80::1d9a:1f1b:f3b0:a33e]) by AM4PR07MB3490.eurprd07.prod.outlook.com ([fe80::1d9a:1f1b:f3b0:a33e%5]) with mapi id 15.20.0797.005; Wed, 16 May 2018 11:25:10 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shapeblue.onmicrosoft.com; s=selector1-shapeblue-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PMeveacuUs0nebIyxaVjnaMzTphPV8opjxAnOcn3gKY=; b=gzq37xKKvYbTxD0GJRtVpLtI4WDyidAo9Va+pV3y6uG8CPZuLZkXVS5xo0Tnttl/lw1QzDDAZQo98/JglylUjhQxd8VsOUjCQqoa3sqLYJ9DLfbcKxET2o89q1lzfFIkQ/v5IWmKeKBzNNDsrVWMSN3Cy19uWQfdMIQJsX5qRlc= From: Rohit Yadav To: "dev@cloudstack.apache.org" Subject: Re: [DISCUSS][ASK] Should agent wait for pending tasks on (mgmt server) disconnection? Thread-Topic: [DISCUSS][ASK] Should agent wait for pending tasks on (mgmt server) disconnection? Thread-Index: AQHT6RXGf7ebwDaR20GcjdbEsqf2mqQsWrGAgAKyGQCAAB5uA4AAJC0AgABDTwCAAPKjMYABpCyDgAANtwCAAAWnvg== Date: Wed, 16 May 2018 11:25:10 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US, en-IN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=rohit.yadav@shapeblue.com; x-originating-ip: [122.162.6.159] x-ms-publictraffictype: Email X-Microsoft-Exchange-Diagnostics-untrusted: 1;AM4PR07MB3172;7:aTjs9bBD/arfNS9PNKqoHroOwqZcpoCZE/vOrm1tgPtmC21xiSMF2awK3EIBxXhFoH3NAi40dcPUmmK82GFs/iUpdPHT8twz2Vy88UoYFf8hfuqt0U1tZ5ALuZIqX87sKxY0CDEgAKpg/cRQs9oByZbH650hubmemaiZS+PfK8Fkt8duqz1v1YlzBzM8G7hVx+BJSxZbeW29UzEqOMn/4AmLxD/xZBdCtxv7szVApWG80RanSu0ay+YHf+fLUvBW x-ms-exchange-antispam-srfa-diagnostics: SOS; X-Microsoft-Antispam-Untrusted: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(7021125)(5600026)(4534165)(7022125)(4603075)(4627221)(201702281549075)(7048125)(7024125)(7027125)(7028125)(7023125)(2017052603328)(7153060)(7193020);SRVR:AM4PR07MB3172; X-MS-TrafficTypeDiagnostic: AM4PR07MB3172:|HE1PR0702MB3691: X-Microsoft-Antispam-PRVS: x-exchange-antispam-report-test: UriScan:(60795455431006)(158342451672863)(166708455590820)(209352067349851)(85827821059158)(21532816269658);UriScan:(60795455431006)(158342451672863)(166708455590820)(209352067349851)(85827821059158)(21532816269658); X-MS-Exchange-SenderADCheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(93006095)(93001095)(3231254)(944501410)(52105095)(149027)(150027)(6041310)(20161123560045)(20161123558120)(20161123564045)(2016111802025)(20161123562045)(6072148)(6043046)(201708071742011);SRVR:AM4PR07MB3172;BCL:0;PCL:0;RULEID:;SRVR:AM4PR07MB3172;BCL:0;PCL:0;RULEID:(6040522)(2401047)(8121501046)(5005006)(93006095)(93003095)(3002001)(10201501046)(3231254)(944501410)(52105095)(149027)(150027)(6041310)(20161123564045)(20161123562045)(20161123560045)(20161123558120)(2016111802025)(6072148)(6043046)(201708071742011);SRVR:HE1PR0702MB3691;BCL:0;PCL:0;RULEID:;SRVR:HE1PR0702MB3691; x-forefront-prvs: 0674DC6DD3 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM;SFS:(10019020)(39380400002)(396003)(346002)(376002)(39830400003)(366004)(189003)(199004)(51444003)(478600001)(19627405001)(236005)(6306002)(105586002)(106356001)(25786009)(74316002)(66066001)(606006)(55016002)(6246003)(53386004)(54896002)(53936002)(966005)(33656002)(2351001)(9686003)(14454004)(7696005)(2900100001)(7736002)(76176011)(99286004)(5660300001)(102836004)(53546011)(68736007)(186003)(6506007)(6916009)(59450400001)(26005)(316002)(5640700003)(86362001)(93886005)(5250100002)(6606003)(2501003)(6116002)(3846002)(476003)(2906002)(81166006)(229853002)(81156014)(1730700003)(3660700001)(3280700002)(97736004)(446003)(15974865002)(8676002)(8936002)(6436002)(486006)(44832011)(1680700002)(11346002);DIR:OUT;SFP:1102;SCL:1;SRVR:AM4PR07MB3172;H:AM4PR07MB3490.eurprd07.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: shapeblue.com does not designate permitted sender hosts) X-Microsoft-Antispam-Message-Info-Original: ElHy8h1Rcksj1DG9/24FVH/DokLcS0PXlpAEQ6ooX/k0iAlXCZ620E4dXEs65QwB+b0dGPUwqSPlBAOhcdvacQx5JCXc2wUmEqbHV+Q1xHrZzcMA200SL4e6w7638CBl7Hf4mNMBgo0Z10VF8ca5xhICPmtWfQtYZxS6wzveRctd4F8R5hUreiAh8hLD/RJk SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-CWesigProcessed: Y X-MAIL_SIG_VERSION: 4.0.2.4454 X-MAIL_SIG_SERVER: smtpworker-in-13.xware-eu-1.o365.crossware.co.nz X-MAIL_SIG_CONFIGNAME: Plain Text for Mailing Lists etc X-MAIL_SIG_CONFIGNAMEPLIED: Plain Text for Mailing Lists etc Content-Type: multipart/alternative; boundary="_000_AM4PR07MB3490B75BB8DEB34B4D74E3F5E9920AM4PR07MB3490eurp_" MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 5ebba361-cba5-4e55-23f3-08d5bb1fb171 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR07MB3172 X-EOPAttributedMessage: 0 X-CrossPremisesHeadersPromoted: DB5EUR01FT054.eop-EUR01.prod.protection.outlook.com X-CrossPremisesHeadersFiltered: DB5EUR01FT054.eop-EUR01.prod.protection.outlook.com X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR01FT054.eop-EUR01.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:104.40.179.195;IPV:NLI;CTRY:;EFV:NLI;SFV:NSPM;SFS:(10019020)(39830400003)(376002)(396003)(39380400002)(346002)(2970300002)(1110001)(1109001)(339900001)(199004)(189003)(51444003)(25786009)(6506007)(16586007)(55236004)(53546011)(6246003)(8936002)(53386004)(5250100002)(356003)(97736004)(3846002)(76176011)(186003)(2501003)(6116002)(33656002)(81166006)(26005)(106466001)(81156014)(102836004)(1730700003)(2351001)(19627405001)(59450400001)(8676002)(15974865002)(229853002)(5660300001)(316002)(1680700002)(44832011)(606006)(476003)(14454004)(61614004)(11346002)(6916009)(486006)(126002)(7736002)(2900100001)(5640700003)(85426001)(7696005)(84326002)(86362001)(99286004)(105606002)(68736007)(446003)(66066001)(74316002)(2906002)(478600001)(9686003)(53946003)(336012)(966005)(53936002)(93886005)(55016002)(6306002)(54896002)(236005);DIR:OUT;SFP:1102;SCL:1;SRVR:HE1PR0702MB3691;H:smtpworker-in-13.xware-eu-1.o365.crossware.co.nz;FPR:;SPF:Fail;LANG:en;PTR:InfoDomainNonexistent;A:1;MX:3; X-Microsoft-Exchange-Diagnostics: 1;DB5EUR01FT054;1:6F6gZOMUFkon3YTl8tlYBekzMepD73qRtnTQcjeihOWXYBYwvdvff1uXJIVZRYwLa10Btlrw1ZhFCeHoseg0HvmDLQts0BqikvMLn5UmNUxSZ02lCIhwckDfAKp6ujcN X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(7021125)(5600026)(4534165)(7022125)(4603075)(7168020)(4627221)(201702281549075)(7048125)(7024125)(7027125)(7028125)(7023125)(2017052603328)(7153060)(7193020);SRVR:HE1PR0702MB3691; X-Microsoft-Exchange-Diagnostics: 1;HE1PR0702MB3691;3:JjiVlIVpA0xv7ER5xmHPp0UUDuYrul3B3jilDfid/NMNrsKau3dLuRIVGSrSgiX5e10UBNmfCh/9Sqfpy8PIBeY8xeE/9Djxjjqepy+CI+R6XBVhDAREyNsGRW9/YiJyn3qFV7gBBbGobYCDOm5KO+1XP+6BF5tYChple196pUzt07CNycV6yfoPdFnnYXhURS5rtVjkU9gWCv+V49wcRhQJXSq7r19x1nrliNYvhy9pyun1/NSP+r0cjV9q/wy4xqi7laT6ZirwbQixxqKpSIGM9YEKtoVCc+kY/T2MHycySl92mR44+NBy00YftuROG+aweuEeVN1QQUrMoYOmErSOxlW6oCIy6Z4pNjHcTx8=;25:vWDgPoVZNsYVkr16mUiNZMRQdXGiOT3kzdhV5XbuCnVM8r+XOBB4JLyJAZ4NxJkC9PD2hL3fLV7I2YTXHdceOA/05TUz6HinKjuGuzPI5bJFCBfcsOIzaK4SV0WJGl3o+YTkfTC3t4XCB0KMOsQGGJUmwO7aZbv04d3rCPJ6sUDkLFBygnlQorUKnfw97kKEMVGTkclmVs/dN0jHXhkJdXdY3DVlSbJTm4KmP3KbTuD45RaX5IiWRsqw3f6EoC7Bu3AD8/qHnIdrxuUxj1SGoM7AlDebu5jgw6ZgJcZDx/6lEjtaZSjbZMT6wIoxMXha3wXo1OphnM3PsL9bJRhjWw== X-Microsoft-Exchange-Diagnostics: 1;HE1PR0702MB3691;31:M03yr3MKEOZ+g2NMHWQwT/ORZZ2/4F2n6KbsmBnSQ0Rs2Jh3ftW+vCAGrkY0QV3t2kNU+K0OXOit+sLUfhg5EsRQck0LINkBpftFN4IvzJOTKtcDfjKZQppw7XQ7vOk9uQdnGaMuA2zQdnBLK19aMfcnvmyw4mD+zj1eoScXzujO5OnlWwpD+cbrYua6DEPPuIiBeBfxXkjhxv4VJttKjIdx/6u1VWMqdhqsJ8VlUuo=;20:TX/1N9WfpkMXUPGoAYGTMN0EwHrwTaD9bzxXVNC5qa1X+3LMfZomV2nVi0unDb82nR/krU9QC8GL+U6AtZPs7Z/NyalKgtoHasvIBzNUPVpxsBGMrlRpdryX/n8IJAy3dS6Ly06+9pgI6lgO9HJvMHmb38dJEzUdRrUHfYnkebhMWf9VYIhQ0YZTJs8WgRWJITxZVN4qS1Qn7gcEJeJhI8BB2Xv3mBe6BUD86pYCfJeC4H+5q+pa+oxFpWRKJLXz X-Microsoft-Exchange-Diagnostics: 1;HE1PR0702MB3691;4:Nht0syWK+ADLhkxcMs9IpdwROIOi/2mgWizPYUsAWAUmz5QXtTu6Qcsnj8q3l1uayCBiUd2xnwASNAyC5NcNMInpBnJYsquw7DyahsPwbzSL550LshcGoXV6vWqShXkrCA0ZBhmOJ04Au90wzq/dRbRy3+S7aKMVpYYxzXZOrpm+qinCJQSpDgazdF6maK5AsmI7g5UX5YQYp5K36eueqXX2SQE4aMnyI8hbp6Fx3U/P0zmYuAznvtkfiHOhXdy5vv15FtkrngrxYUnRc2W7OYMv33RRGVPh/PKEBmbpel5mbHxAz/6uIWaHtVQI2EyPDqll+eASnkhJdvQxQj9dgJsjC11d/jUDxXgNYAJyRwiGzPrDdNWzGYXb2VF3rQh/Lp/aoCdtB5B44xIyEsGI117zkA6PgjcRHkwEvPmPfvQOsyuL2UdZM6lCC9xu7JhisqLUY6S4olLyWTuUnyDRIKh6xBHPOLDWTY7F5seLviY= X-Forefront-PRVS: 0674DC6DD3 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;HE1PR0702MB3691;23:uqx6x1UIRPM3fX1UcGySYmab/GkW9dzNVADdvi3?= =?us-ascii?Q?7QSJsd4+74SO8wnePDkXmR5hbuHipbyY7hU3AYXocyq/jFgjwEoF7YZJ9lCW?= =?us-ascii?Q?7thRestBGlDZNP80x+yV5Xr3ivBSk5HSVzbFBPsDb4Me9QublcPt2J3uiAGn?= =?us-ascii?Q?Yo9lU6sR6u6VltKXvw2BDEyJMq75QjtIh11L1LUdXa+en0I8esUMrFFMaVMg?= =?us-ascii?Q?f0MHQMMsEBX3zkpCflADf5ymHg3WRlEwAdUHjrGFY7Ji1MfgwguBpiIYYgkk?= =?us-ascii?Q?gXf7QdZqDTVEG2cM+8LTdtto4QB5jl/lKbx9aTK5LkWHA+LX1TA6vVhG6IDT?= =?us-ascii?Q?C7t1FgQJ05Tt50acz+idnaR1sDV79PDOa/R5vzDpSRwmTyitGrlBxSad/XCq?= =?us-ascii?Q?RnmKS5UELHyrftlCXv6e7nuGFnqoiWZ0pwcolUiJ8MPPou0ccrXz1dUGdBIb?= =?us-ascii?Q?R56KRe/EteixStYMQwSnHGNtbZqDzoEevPXDOH62wtWFAYLHRA1JBCsgdel1?= =?us-ascii?Q?VruxOkYAn2vVKYJniUTU+dodZfTJK5hpt5SMMhR2JwcqcNw6+7RvoL9iI0xo?= =?us-ascii?Q?gYM73Zt9a9254hRXcxPioD17wNE+VKrBJaeMULHJnPCzKjoYuzTfG/Ih6krS?= =?us-ascii?Q?adZGt1OrJInbxoxLgtwgrfJdXMUt3wLRsArEQRcGkboXkXI7Xpn4MZzxe9hA?= =?us-ascii?Q?I05HQkLoYo2CVoSPc1bdihTfrAJ8HorujM84XbdI4wPrsQ0K0boXgS5gXUz6?= =?us-ascii?Q?1zD4Zs0NWxOdyIeL7Xzjj+N0QJNdVyHtOqzFhYUltjBsxE3xBCDE012a/xLl?= =?us-ascii?Q?emOOncTAZzF5pTfemWaOm8SSqqp4s5dfrFHF8XT3PyBP1mvJsGAV/48xA33J?= =?us-ascii?Q?SJhm/W3rMvMmPvylhL3nGCzBnNOlMy7u9IIZcM9eZQLacWUtIZNRvAsdD8lB?= =?us-ascii?Q?AyHsAtsAsgd8TubqGtdX82EDxobGqX6eTQszDn9OCOkme9UfX84JTSZEVz6K?= =?us-ascii?Q?ILeiMFTngb0FFaP9Rl5Ho7v1718hlK4bllTQCd2LAB2/vb2yIiOIpRrmdq3I?= =?us-ascii?Q?X+xufRN2brlmnbjWN1QWtlvN/05vrsoHc1ImmmwQU0i8qht59sVMvM01m6Lp?= =?us-ascii?Q?AKfFPvqz++VxHkfa73ropmo+IKsFRqDW0fMqYrUP1t7gySgaiAAe2rYneSMK?= =?us-ascii?Q?4Omk7KDS708RS03d/UQdUHDR0ibI+s4Yes1CLol+vgXhIG/5zPHVox5CXXha?= =?us-ascii?Q?s35ImBnByUXSUrPZ0ZxCvrWl8+/jCmqbohGwllGYTYPsL6Iy810hcMhuaL1m?= =?us-ascii?Q?Y8JhpH2DBN0M0fqkyW2pNcp1JPvgE3gl5DIB0TUaNbEjBkAOYI30H2AlE6Bs?= =?us-ascii?Q?5qvAEFWi4ylY3jqi+TeKEO8sNS9gVAzVVt/wgCacmOSwWNgi3TrwR4a52f/m?= =?us-ascii?Q?RLmjip1PrsazwA5llTAaOqqHzCLdFDZLRN3zBzWsw8QwFaPvfHIT0VCO3rcv?= =?us-ascii?Q?ynxcylBI3SsP/CFP1uOwOVwvWTGtpxtZe/X1CImBLcT/tLAuuQ5CHIsVagzz?= =?us-ascii?Q?vXX0q47CUG6YDuX7zfl4A9jjATVUhKAIBMWTDdX7ccAng2pvTxIrBrHM+Als?= =?us-ascii?Q?hYimq8ygavCdBMPejtjMsOSW00FIGhKbSlNRPTIa9sIXicnzI2FmUMXjnSa9?= =?us-ascii?Q?+Ht9a4vn8aiyXKmzFIMJLSmh01rFMbaIDFiCv/0sjXVW8mTNV6HvvPfqy5Oo?= =?us-ascii?Q?FCeQepZV7?= X-Microsoft-Antispam-Message-Info: dhS+1Px440YV5KSwRVfLtm2W7rKQQO6mvlXQC8SUAk+YqKL8hllbQg5CcucrioKqFTFMl1VRdPhoLcc3Lm52X14lRLGcmU4YU+z4Igs5jDVvCk7FfIVB1ca6nmVlDLttxt51fz0Q6Xi9VDkbTdiZBdYt6L0DF6TzPJA1cV0fCTPWI4H2UGEdO7TQuqTvfcIz X-Microsoft-Exchange-Diagnostics: 1;HE1PR0702MB3691;6:QWs1NyaAsnpwVRbIrVg6y0mAvkLB8Rm8VxWnCpFg7UFiaT63A0CcGOLdgfAh4FqDoM0unnZOMlRNq6IgZe6Q8Ef3JC9LYYBkO6h4WfwNv6HHgeZK6a5fjQVReYRCxDXTvLZePKzKkeKgKNvLo8zLRttx0+RPgab7uioo8NGkxHNTAk8yHhQ3soUWJlJvBOVKwaMzfbTJ2SUvid/TNc7gqGaePpHWlyKxsvGVrIrMwVPaJZ+9CWHTjQ8Z9md/TIjkkLb+e9OfnZv/UDDyb4u9vhvcBQtyl1efeeogXtYFnpxIidpm2rSGTUyXuHmWcZJl95G1mBD2Gx0HSbDrXw+Z42F751lGQmhSO43VvrsRZEQQ7rGt6V2hJj/hFcipGjZbeHQ+6nSwkh5wEEdsTqAXNL1VFIVUB74idUgUz1HjOHDoZ7JdZRG2JF8RjlHeDahfjZTvqwSQEfl5CbeVMBgLvw==;5:+EQw8GyWFRo/D5h8ltyFa1eaWy446g1Pv8KDYmzXuSIg6O5CFroCYio1LlqpUhMcXr30hoVbyfi1z8xXtCvQ7YPOcYH6x7zgngLDBHMvkRYaXye1PFBsuE99xY8ImxaPawdIkexZPMvfKlq7RFXVwgmLuYBwQHCpzEiAq6vahHs=;24:LaOIc3ac37OUW2PgiEKK9s14xzI42Ns90xK6XPqBVUliKYI+szURPIWSYHc4xP8teCbP48FmFkAb9AtFCGSMFWyIrNmLmVJ2q4Hw/IBZ0H8= X-Microsoft-Exchange-Diagnostics: 1;HE1PR0702MB3691;7:k9SWZzSt9lwFAzdg6Ek5RoY3yalqkwPnWD8S2toB4HXBB7FtRTBP3Q9cGyBaAZ3sFbOHPKi2/PG7wp8o+qMkELqERZAhUGs96cXgF8Y1/pH8zbSfA5Ok/gmo90Q2L/GIyrG00Qb6FXYkak9RRtPrXJxXh8mXsbguWQHdAfArOIAZ4t155Ej2pxs773H7EMkLoekvoWabgKSjntKKK00Jz+3ltHBbsiHG1TOH8GlbZFhQ2NiAOHt0ikDA9F65uNGr X-OriginatorOrg: shapeblue.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 May 2018 11:25:13.7530 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5ebba361-cba5-4e55-23f3-08d5bb1fb171 X-MS-Exchange-CrossTenant-Id: fc8906f6-e50e-4dad-98a0-ec2e3abe14f5 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fc8906f6-e50e-4dad-98a0-ec2e3abe14f5;Ip=[104.40.179.195];Helo=[smtpworker-in-13.xware-eu-1.o365.crossware.co.nz] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0702MB3691 --_000_AM4PR07MB3490B75BB8DEB34B4D74E3F5E9920AM4PR07MB3490eurp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Suresh, As explained earlier and advised to look at code on the PR, perhaps you did= not get time so have a look here: https://github.com/apache/cloudstack/blob/4.11/agent/src/com/cloud/agent/Ag= ent.java#L488 The reconnect() historically sets the link to null. Therefore, any answer f= rom pending tasks end up failing here: https://github.com/apache/cloudstack/blob/4.11/agent/src/com/cloud/agent/Ag= ent.java#L868 and, https://github.com/apache/cloudstack/blob/4.11/agent/src/com/cloud/agent/Ag= ent.java#L893 Do note that reconnect() only cancels watch tasks but does not cancel/shutd= own any running task. Also, in case of network error, the mgmt server will = fail at thread/context where is has done a agent.send() and expecting an an= swer. You can also perform a small test by doing a while or sleep around this cod= e to see how getLink().send() behave when agent does reconnect. When it doe= s not reconnect, i.e. the agent is blocked by pending tasks to complete suc= h tasks always fail. - Rohit ________________________________ From: Suresh Kumar Anaparti Sent: Wednesday, May 16, 2018 4:27:36 PM To: dev@cloudstack.apache.org Subject: Re: [DISCUSS][ASK] Should agent wait for pending tasks on (mgmt se= rver) disconnection? Hi Rohit, When Management Server and Agent are up and running and there is a network failure, I think it is better to wait for some time for the pending tasks to complete, instead of failing them and try reconnecting. If network delay is minimal, there can be a valid thread/context in the management server to handle the answers. It would be great if there are no major side-effects with this PR changes. Thanks, Suresh On Wed, May 16, 2018 at 3:40 PM, Rohit Yadav wrote: > All, > > > Based on testing against KVM, XenServer and VMware and this discussion, > I'll merged the PR based on code reviews and tests. I investigated both > code-wise and against live environment for possible side-effects of letti= ng > agent connect without being blocked on pending tasks and I found no new > fault behaviour. > > > If there are any objections or bugs, please share in which case we'll > revert the change to continue legacy/historic behaviour. Thanks. > > > - Rohit > > > > > > ________________________________ > From: Rohit Yadav > Sent: Tuesday, May 15, 2018 2:37:58 PM > To: dev@cloudstack.apache.org > Subject: Re: [DISCUSS][ASK] Should agent wait for pending tasks on (mgmt > server) disconnection? > > Hi Suresh, > > > I've replied to your comment on the PR. In addition, when (i) management > server is restarted any pending operation on KVM/SSVM agent side will fai= l > fail to be communicated back in the correct thread/context and it depends > on a specific feature whether is supports sync or cleanup mechanism, in > most cases, the async/job timeout may kick in or cause queue/concurrent > failure seen in logs. When (ii) agent is reconnected, it reconnects only > after any pending job finishes therefore such jobs finish and fail to be > communicated back to the mgmt server (the answer instance is failed to be > sent on the link, as link is no longer valid and causes exception). > > > - Rohit > > > > > > ________________________________ > From: Suresh Kumar Anaparti > Sent: Tuesday, May 15, 2018 12:06:14 AM > To: dev@cloudstack.apache.org > Subject: Re: [DISCUSS][ASK] Should agent wait for pending tasks on (mgmt > server) disconnection? > > Hi, > > @rhtyd, I checked the PR changes. Good that the agent is not waiting for > the pending jobs and retrying connection to management server. This might > have impact on ssvm and kvm agent tasks, not much on cpvm. Any sync or > cleanup mechanism for Volumes/VMs to address the failed/pending agent job= s > after (i) management server restart and (ii) agent connected ? > > -Suresh > > On Mon, May 14, 2018 at 8:05 PM, Marc-Aur=E8le Brothier > wrote: > > > Correct about the thread context, so if the answer is coming into a > > management server that doesn't have the context and drops it, it should > be > > fine then. The PR is then already a good improvement to let the agent > > reconnect even when it's doing a long processing request, so it can kee= ps > > on completing other jobs too. > > > > Regarding the restart/shutdown operation, yes I have to push now the > > changes to be able to stop some processing tasks (fetching new async jo= bs > > mainly) on a management server to ensure a cleaner shutdown. My solutio= n, > > as said, is based on the content of a file that is compatible with HA > > proxy, thus not the LB mechanism added recently in CS. It could be > changed > > for an API call to put/move out a management server from maintenance. T= he > > listManagementServers API call has been merged and it was a requirement > for > > that. > > > > About Zookeeper, it's not on the rolling shutdown/restart for now. We a= re > > using it as an efficient and true lock mechanism between multiple > > management servers. We are slowly moving the locks code towards ZK and > > added one during the allocation phase to ensure no host would be over > > allocated. I will take this discussion in another email threads since I > > have a few questions regarding ZK and also which to talk about the > > connection between the agent & management servers. > > > > On Mon, May 14, 2018 at 2:39 PM, Rohit Yadav > > wrote: > > > > > Thanks Marc and Rafael for replying. > > > > > > > > > In my experimentation, when agent disconnects if will wait for the > > pending > > > jobs/task to complete and on completion it creates an Answer instance > and > > > tries to sent it using a `link` which no longer exists and fails. Thi= s > is > > > current behaviour, on the mgmt server side the resource/task will be > left > > > hanging and may not be automatically marked failed right away (may be > > after > > > the configured timeout). My best guess is that the application of the > > > change should likely not have any side-effects, other than the > > > exceptions/faults we already observe. > > > > > > > > > In my test, the failed async job did not get retried and I hit the > famour > > > 'concurrency limit 1' issue. At this point, I had to manually cleanup > the > > > snapshot row, the rows from sync_queue, sync_queue_item and async_job= . > > The > > > current implementation we have on the agent side where mgmt server > send a > > > cmd and agent returns an answer after processing it -- we don't have > the > > > same for mgmt server where an agent sends a cmd's answer and mgmt > server > > > processes it irrespective of the context. Therefore, unless the answe= r > > > receiving mgmt server is not in the right thread/context/state those > > > answers are dropped. > > > > > > > > > I think we need to solve for (1) claim and ownership management of a > > > resource (how to manage when the owner/mgmt server shuts down or dies= ), > > (2) > > > task handover - executing tasks (in-flight) when mgmt server is > shutdown > > to > > > other mgmt server, (3) central locking-service for this and other use= s. > > The > > > bigger change ties with the other things we've seen in the discussion > > > around mgmt server restart/shutdown. Till the time we get to solving > the > > > bigger issue, perhaps we can provide some API/visual/UI ways to show > the > > > root admin the async jobs in flight for a management server or alert > him, > > > perhaps an API to do cleaner mgmt server shutdown that waits for all > > > pending async jobs on a mgmg server to complete and does not take any > new > > > async/job API requests (say like Jenkins does with jobs)? > > > > > > > > > Marc - were n't you working on a zookeeper based rolling > > shutdown/restart? > > > Did that handle some of the failure cases? > > > > > > > > > - Rohit > > > > > > > > > > > > > > > > > > ________________________________ > > > From: Marc-Aur=E8le Brothier > > > Sent: Monday, May 14, 2018 4:06:56 PM > > > To: dev@cloudstack.apache.org > > > Subject: Re: [DISCUSS][ASK] Should agent wait for pending tasks on > (mgmt > > > server) disconnection? > > > > > > Hi, > > > > > > I'm also for a bigger change but this PR already moves forward to a > > better > > > agent <-> management connection hanlding. > > > > > > @rhtyd did you test your PR manually by, for example, requesting a lo= ng > > > snapshot operation and disconnecting the agent. > > > > > > I have one concern here: when an async job is taken from the DB by a > > > management server (in a cluster configuration), the mgmgt ID is put i= n > > the > > > row to tell which mgmt is managing the job. On disconnection from an > > agent, > > > the event is propagated and the job is mark as failed in the database= , > > and > > > an error is return in the API for that command. Here we are only > > resolving > > > the fact to let the agent reconnect quickly but I'm unsure of what wi= ll > > > happen in the mgmt when the job response is received by a mgmt (which > > might > > > be another one than the one registered in the job db row). I know it'= s > > here > > > it's becoming complicated because one async job might be only one par= t > > of a > > > bigger scenario for a command (like a live migration). I just want to > > > ensure it won't propagate further inconsistency. > > > > > > Marco > > > > > > On Sat, May 12, 2018 at 7:26 PM, Rafael Weing=E4rtner < > > > rafaelweingartner@gmail.com> wrote: > > > > > > > Would prefer =93A bigger design fix would be to make management ser= ver > > > > asynchronous of agent side answer/response handling=94. However, I > > > understand > > > > the volume of changes that requires. > > > > > > > > I looked at the PR, and I think that everything is ok there. Of > > course, I > > > > think we might need some more time to review and think about the > > possible > > > > outcomes of such changes. > > > > > > > > On Fri, May 11, 2018 at 7:55 AM, Rohit Yadav < > > rohit.yadav@shapeblue.com> > > > > wrote: > > > > > > > > > All, > > > > > > > > > > > > > > > Historically, when the agent (kvm, ssvm, cpvm) is disconnected fr= om > > the > > > > > management server (say due to mgmt server restart etc), the > > > reconnection > > > > > logic waits for any pending tasks/commands to complete before > > > > reconnection > > > > > attempts are made. I tried to search git history but could not > find a > > > > > reason, can anyone share why we may need this? > > > > > > > > > > > > > > > Based on the reported issue: > > > > > > > > > > https://github.com/apache/cloudstack/issues/2633 > > > > > > > > > > > > > > > I've a working patch which removes this limitation: > > > > > > > > > > https://github.com/apache/cloudstack/pull/2638 > > > > > > > > > > > > > > > From testing with various combinations of tasks, I found that whe= n > > that > > > > > happens even if the pending task succeeds it fails to send an > Answer > > to > > > > the > > > > > mgmt server, therefore from the control plane's perspective that > task > > > is > > > > > still pending/on-going. > > > > > > > > > > > > > > > When the mgmt server comes back online, and the agent finally > > > reconnects > > > > > (pending on how long the pending task took) the executed operatio= n > is > > > > still > > > > > pending in mgmt server's view and may sometimes require manual > > cleanups > > > > in > > > > > database. By removing the limitation in above PR, at least the > agent > > > > > reconnects faster while of the failure/fault behaviours remain th= e > > > same. > > > > A > > > > > bigger design fix would be to make management server asynchronous > of > > > > agent > > > > > side answer/response handling. > > > > > > > > > > > > > > > - Rohit > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > rohit.yadav@shapeblue.com > > > > > www.shapeblue.com > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > @shapeblue > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Rafael Weing=E4rtner > > > > > > > > > > rohit.yadav@shapeblue.com > > > www.shapeblue.com > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > @shapeblue > > > > > > > > > > > > > > > > rohit.yadav@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > rohit.yadav@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > rohit.yadav@shapeblue.com=A0 www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue =20 =20 --_000_AM4PR07MB3490B75BB8DEB34B4D74E3F5E9920AM4PR07MB3490eurp_--