Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id DF3E1200CA8 for ; Wed, 31 May 2017 19:56:23 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id DD56F160BC2; Wed, 31 May 2017 17:56:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3E490160BCB for ; Wed, 31 May 2017 19:56:23 +0200 (CEST) Received: (qmail 61642 invoked by uid 500); 31 May 2017 17:56:22 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 61480 invoked by uid 99); 31 May 2017 17:56:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 May 2017 17:56:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id F1B33180158 for ; Wed, 31 May 2017 17:56:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.022 X-Spam-Level: X-Spam-Status: No, score=-0.022 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=rmanet.onmicrosoft.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id PECRoq5IYVh1 for ; Wed, 31 May 2017 17:56:18 +0000 (UTC) Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0106.outbound.protection.outlook.com [104.47.32.106]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 719AB5F5B4 for ; Wed, 31 May 2017 17:56:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rmanet.onmicrosoft.com; s=selector1-rmanet-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=b+G7Vnhj0hESdp5p0VFrVgPgsi8XrIe0SMAQlaTGaOw=; b=C9i9v8N62lUii/0vY94/yLLVGsmlAWth3oMbHjF+oOf9pnLtxIF/tRXm5cN7vVMyAbDVsftPLLPNPqt/DaZ3xN8GsTrFwDZTGuW2x1zPqGdg1L8Xfmrp/aGSRlWa5MYL4w8Vv4TPTmpX+Uh64KAcREUwLpFBdgX0QwlEHip9AKQ= Authentication-Results: ignite.apache.org; dkim=none (message not signed) header.d=none;ignite.apache.org; dmarc=none action=none header.from=rmanet.com; Received: from [10.0.4.10] (50.193.37.125) by BN6PR02MB2321.namprd02.prod.outlook.com (10.168.254.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1124.9; Wed, 31 May 2017 17:56:10 +0000 To: user@ignite.apache.org From: Ryan Ripken Subject: Suggested logging settings to debug disconnects? Message-ID: <128c980b-6a71-23ba-933e-1a156a98948b@rmanet.com> Date: Wed, 31 May 2017 10:56:07 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Originating-IP: [50.193.37.125] X-ClientProxiedBy: MWHPR15CA0056.namprd15.prod.outlook.com (10.174.254.18) To BN6PR02MB2321.namprd02.prod.outlook.com (10.168.254.11) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR02MB2321: X-MS-Office365-Filtering-Correlation-Id: caf9d5c0-f52e-46a7-8e6c-08d4a84e525a X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:BN6PR02MB2321; X-Microsoft-Exchange-Diagnostics: 1;BN6PR02MB2321;3:xHo/bZgyEIWVaz3oDghLtEHPkHBGq/2ERjBMIU7s2X8QAK0zU08nEdh0V+/FnrFw7WyNo+Ls352myPHNkeBuUHfT1nOYJjqB8IxbvlOOLuecEJ8ZHw/ynTg/S6KVpiy9aJsNqqP/bL3I2vsyGP76Whdw1Qpj5mvEkIOF8m2JxfZSA9saYKieTHPKr8snnCGz/37CxVEhxkEQ9BSJpFnugMotiXY4sTSBEbi6rSWdq8MhhfslRIP391Trgx8zt+nfMt8suiKN8KgLguzbDMFlgYpkG0UJcsr563iYREbO/7A4Ffh2KNwfETfp6ZmZSpCxoyUsL567gnQuI4earAR7pw==;25:Ca3ztjOQIzH5MCqtT11FphZAelR/4L6AFqMBAsjyYUKnsK7lAEWcZxRlx6RHy7G/+SsHHDK29uWP4BE7qqHfkG9u8Orh5LQ8m8P4CwJ3fldJL4edfH+AIJ5wUIlYe5WczGn/IZtDLhMUSMXLlU9qMk3EpH5gfIyQ8bt/wjXSclXG637NWqo7HW2oVPCG7BXi7SEone+op3QAis6Qe03+6k218k6u6Zf6R7XIn4pl9dWNSbvbYamTts/lDiBZRVD6IXSz20U4pUT3PhY8emR6IZhsvu1AhORHrTXhqOgo/GrPxmy6p4aVO9wJ0+UcD9kzY5I3VGTj3daTEk1InPLnz5uuotV1wIdy5UjsZSBuj6EOnz+3yEcQWoRHI1qJMexUf5mHoeihqId/iMZc8aR16VYo6AUI3yzLYiBbR6wkEjhhgMZRL9Y078QIgLGRtOmyO+iJcCAPsd2HyzjUO80l/StwIlWRosOqmhQ4Ru+tMeI= X-Microsoft-Exchange-Diagnostics: 1;BN6PR02MB2321;31:Rzf+Q4zYY3dHQ86Z+RNHuQGd/DGXTLEfAIH0fz2+lKOBH07S94IkMXJWZBDkn2gbJs9rprGl+mB731SyUi3HVGquGGm2TB8UJCPwShy7XHa44BR6Z2lCBJztcJJc50z2MlfGSLTYLV8gf6zitgaN+atZjs7Dqx2j2FepL4Fpgw7gKKp0cn8msWoeCLgoK21De3WvAtmhOJXJBgp1eqhCxvSowiMYp7sCJ6T/+A8qaoQr6dzR3Q1iSK60OCVZSX1Cik+xhS5VYSLoSKmOLcclow== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700073)(100105000095)(100000701073)(100105300095)(100000702073)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(100000703073)(100105400095)(10201501046)(3002001)(93006095)(93001095)(6041248)(20161123558100)(20161123555025)(20161123562025)(201703131423075)(201703061421075)(20161123564025)(20161123560025)(6072148)(100000704073)(100105200095)(100000705073)(100105500095);SRVR:BN6PR02MB2321;BCL:0;PCL:0;RULEID:(100000800073)(100110000095)(100000801073)(100110300095)(100000802073)(100110100095)(100000803073)(100110400095)(100000804073)(100110200095)(100000805073)(100110500095);SRVR:BN6PR02MB2321; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjAyTUIyMzIxOzQ6QzBtcGxXNld2NWhDY2U5M1VCZ0lxZ1JuZ01u?= =?utf-8?B?NHE4Qmk5L0ljQUVSTjhua2xpdVB1eXVxTTZJOTllREdnUDY0NDJubUtyV1l2?= =?utf-8?B?TE1IV2ZPbUs5YjRLYklvaS94THMrYWJGVjBtNHUwb0gvTUtUZGNRTjArNVl0?= =?utf-8?B?cHRSdzIvcDBWWDUxWXlScXNhMTRIQk9uNzNDVnZ5cGRwdUdhUHFSdXBUOExw?= =?utf-8?B?cEdKNSt0OVlqaHJSVGxkRVVDbmJiajlsaUtCN3AyeHg1Qnp6KzRidHk2V3Jz?= =?utf-8?B?RHF0WU02NmdvK1FwZTljaTJ3MEdCMDdEbkkxaDVDcXNac25oUWJ4elg0YXla?= =?utf-8?B?dVAwbzlvbVM4MHNSampTdlJZYjJ3VVR3eVVsK0h6eHY3ZDFWSDFSWlZ5bEp3?= =?utf-8?B?anRzK2pkTW1PQWxkdFkxVjZDbDQyd05VaGZVR01PbE5lRVFjV1NidlVoY2lp?= =?utf-8?B?aDdaL3BxTUNpbjFobHA0VDhjc09rYlpSRk0wUFdtT0pFNitGckFTc1dwSGdP?= =?utf-8?B?WGxuVU5DYlh6aWUybVFwejlCSlR4WTRVTmVrQU9CS0NwaTZhbVRva2RqbllK?= =?utf-8?B?Nmc5M0VsUFlUcEowbTlDbXFGRjhzS0NyUVdqM0pXMWQrZFF5QzNBWnNVQmdv?= =?utf-8?B?SFRYbWFDc0FuVGFMcTc5YjRtbGZPcG1VWVppSjhKRXNYNnVzTHp6RWkreXdD?= =?utf-8?B?a3Y4dVpNZ2w4UWQzNTBTVFZ3VjJtU1h5d1hmV2J1aWg2QUZhcFJXaGh1alkv?= =?utf-8?B?QUozbVZmMzVsSlF0QzdSTDk1OVFmcldIV1lpSlpIUk9CZHRSOFlPVFAxb21S?= =?utf-8?B?b0c0WkNoRGd1b0QreUcyV1ZLNEliRnQwKy95Y3VNUG1IbkpLeXdrRURhSlZM?= =?utf-8?B?QVFwSStURGRUK1hUcnU4NFkyNGNTTG9Ba09JS3N4VytrMGRhTy9PTkpnbHhL?= =?utf-8?B?bU5nVUJ6WmRwb1hsWFBwZ1ZlbGtMdjdJN3VkblpwRXEwVWtoRTU3VGpESlFq?= =?utf-8?B?NW1td0ZLNmNKVEF2T0RxeEluWFpWaGpOeE9CNGVtRGJramxYYllQcC9zUFNJ?= =?utf-8?B?VDh2RnRTWVhvWk11OHA2SFYvU0xjYVpoRWIweXUyeEhXYkluWVY4UnZMM3BX?= =?utf-8?B?SGVva04rTW5QcXFzMnFXL1l0N0JrQUd5YXhLU09FelhaQW5HUU83UVloOE4y?= =?utf-8?B?UC9sZ3A2RlppZ0pDMEdsVk04OThiaHErY0YwYS9TL0o1bW5iUUlsL3V3bllT?= =?utf-8?B?TCtEem5MZUFoMExmSEhESThCSW5NSTkvQjFxampyVkpDUzhnOFF4Sk9FdHhp?= =?utf-8?B?Snhyc1ZiT01kYVhlM3FmTVA3ZWJpWTdkWms2blFXK25seHhobzJsSHJtMDdY?= =?utf-8?B?OVgyMUdvb250N1VkVjdNSEpDWDVrZjdTN2ZST3luZmJWSEs0cUVrRmdYc0g5?= =?utf-8?B?eDFrWnlXUFNxdlEyWi9zcVpnUDdDcWxPOTdLaHFzdnNneUprTjZyNWp4Wjc1?= =?utf-8?Q?ePoGcpAOf0nblvzupcjW12Dw=3D?= X-Forefront-PRVS: 0324C2C0E2 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(39450400003)(39400400002)(508600001)(50986999)(4001350100001)(8676002)(31686004)(42186005)(54356999)(305945005)(47776003)(50466002)(65956001)(90366009)(77096006)(6486002)(230700001)(81166006)(5660300001)(66066001)(64126003)(6116002)(3846002)(7736002)(110136004)(6666003)(86362001)(33646002)(83506001)(2361001)(38730400002)(3260700006)(2351001)(31696002)(189998001)(36756003)(23676002)(53936002);DIR:OUT;SFP:1102;SCL:1;SRVR:BN6PR02MB2321;H:[10.0.4.10];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTjZQUjAyTUIyMzIxOzIzOmwxMVhnSGQ0NjVraVM0WVdqbFlTbitUSmlq?= =?utf-8?B?SWxwQUdSR1hyTFpPOTJNRWxqRFVabTFXRDMzUkRERzQrVmVnazBNUm8wWVNC?= =?utf-8?B?VkZOaU9YeHQ3MmY1bmk1cFVPY0NZc0h0Y0JXNjhrMUcvNklWWVo5K2JYMktq?= =?utf-8?B?SndPdHVvaFNBODZ6cXBDK2F6VzVKby9NT1pteEhHS1hpTXpGRlVYZDAzNmdL?= =?utf-8?B?QlQyK2swVlhYakd2Ym9IK2xBUDM2T0VBSHNHaTR5V1lpQUIwcDVsVHNxb0pD?= =?utf-8?B?dHVyeHprc2NEM0FxQ0FkMHBjYmcyYUI3QzNkd21aVkFVcVhvNFdoRzlSSUNl?= =?utf-8?B?S2pBeU0xNlh2VExNVWdJaUJkWmVBRDRhTWowdEx1clBTS2taa2Y3RHFhSnl3?= =?utf-8?B?Wk42VU9USnk3WnM1clJXTnNBdFFvV0hMNUFPWkFPbC9ZZHhiY2RicnpvbDJi?= =?utf-8?B?S2kybTdaNE5GcW9pQ2hxYUpUcUZrZjAwQTZSVHJQUkVCUHdvZEVCdFQwZVB2?= =?utf-8?B?RTl6dUY4dUNBaTI4b3hQUEcrekl1NTVWNVRFZTMwNTFMRm9Tc1JraGZyYXVB?= =?utf-8?B?ajhYYW5SSkxtK2IxZXZEUEZ1Vk9CRlgrTlplb2FvMmFrd3pVWGxxV1B0RXR2?= =?utf-8?B?TFEwb0hzMXlTZU1hVkZrU2RMZlFXRG5NT3VaNTc5cXlDWmc0S3Rndjg1SWt2?= =?utf-8?B?bHRPTlVzd25tK0NLYVlUNmVhRjdUUXhmVGJJUVJpMi9VUzd5RjJsQUVKd2l1?= =?utf-8?B?RzlmK0JvWkFlL3UwNmFsZWdJREp2R0lwQ3ljeU5IMHBtNVZ6VDVPenBUM0kv?= =?utf-8?B?SWlWVVFMdHczc1VPNS9kUm9Jb2kwb25Ib1ZPNGlsRS9iMWdUMWJNU05zbm9x?= =?utf-8?B?MkpDQkV4dVJOVXJxUHdSTHhscXlVWU92M2F6cEtma3B6MFFzeUpNYVJFQjlT?= =?utf-8?B?akZ4U3hYRmtTd3gxQkNkeTNiclM2OUZ3RkNjY1RMNFhSV0ZVd2o2ZElHamRJ?= =?utf-8?B?czhCb2o0dkh3Y1pjeldKUEEvalRDMjhyaTFHUmE5WE10T3RnV3grTzFlRFpP?= =?utf-8?B?TTRJUjJvVkVzZzJlVTUxQk1zZEZ4MUFEQm01SUNheGdZd1pOY0dmNzNsTENu?= =?utf-8?B?c1hNcUc5N2hPTGRYR0NKTnE4YWdOcDE0TENaVnA0anBTMUxERnB4YlFaT3NK?= =?utf-8?B?OTNiRXk1OW9ncVhyRVdrdXl4YjlmdGttbTkzdncrWWplR0N0VE1hU1J6OGRl?= =?utf-8?B?R1B2clh2Y25UTEF1ek1GNUZFRTZBcVR4b2tTaXVZWWtxNEZ5U25yaTlrbXFv?= =?utf-8?B?eExQeFRIVm1xeG5GSzVtdVdnTmx0TzU2R0w0U3dhSisrOXNGTnd2bEJTVTQ5?= =?utf-8?B?MW1QcEc3YTVyUm45cnB6aVRaYzNBU2FTZFQ1RHZ3PT0=?= X-Microsoft-Exchange-Diagnostics: 1;BN6PR02MB2321;6:f65SqHHeZkjbTlfSHfyKx+xS8HMtkXX0Qcp/pLSsVXxFJKXdMQZ73dXBx0UdWXC8kj83oXb/31QnoEipNbHTzdxjmJnzcYoL57/AkMBGXUErioeHB5Q6xvlx4XlQJcsepF8BgKG0VqyFA62giIbC9B5u6HP2HlyNQl/PTukldKVP/DqFFFm8xTBNZ07zAL7RqIOLVtnMEKKTym/TFEllkDiqthdMrrtq0OSbY5HUeYxxmpxsyp+XuO450RLQxE3Pv4jECS25CuY0XiX6245JKDuHYSo+x3TeJQJN7U42M7aD/F9u3c70XI4FrhtoBRe3LHba6F6envP6gZjk7ctx0vIhRVU4X9gUXZS0+7xWTLlGOSERVHKl+T6rpnK6WrIzYzHYwJ20W6YnFgubpIsy3mso7AwwxJ3dAoUi5/NJjVcYcqwioqgUem/EefGeSmsIGJViLV84L0jKR9s0fZRLGNV1c3LQwgVAL//N1JULFOQttkIflY7alQEW/csC30gdS2Rkf9lRXgZU1Uy+UIWcuw== X-Microsoft-Exchange-Diagnostics: 1;BN6PR02MB2321;5:D7HUr0LWeV/k5MflNmrxsRD5BrNI2tspyh9RbcfXJuWXPAELFBDgtiryvxZfF7SEYlyxNQgx++oUKSnhdjIBkQ0VWqfFIlqSSmdgi60Gpx2ve9A5h5agCKH4CDQYwP2EnfSWOnNVXg9ov1+7cxAczYWTSGNYbrZxiSIvsAP92baBeuHRX8wqCQZqaI0gQzSwsbZc0VehHuwWOkLN6RdhF4d1KpIAG84KA3choBEJY43iU76E1Ss/8Z7JvOtUJp9lNKI12zracfLN4hYmhtiUY4mq0DappnBEcL59eSzQncf5YHhpwbnsYiTaqoKlz8eyd2c1pCyr0B+nPxJRlznHtF7oMPgmCJZQi9MBFukEjSDTSD8XFRBas1GwcQsq/otHYd4At1GJQHM/mREjm2rvJnFEvX1MH2fHqIs28RdCj5mH79IBXenRenHnhVIKgqcAiwn+gbflolJMuf+cijcerFU/GtxlKnUuKYWDZVlG1kNAJbvuqPHQtCM5GbzA9gxB;24:YsL6fwHpEdHX5kaBLmj2wzRGx46KEeW6cjIQdpJgBoJ4lYPhaDyQkTCp9JAxyCuq6m5rM2o888haQEDIjOq/YrRm9hgOa8zn657vdTJQZlY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BN6PR02MB2321;7:1zCXytXw9iKR13W/mcRra2vVMNrG3DZQhDchARrB0NQjoG6X4j1lodnlByOVDC4X/tYxrUJEdT2YxruiTurXddjM4H1d/sFy5umTcFU5k2e7YhzoI+FAWl9iTPkbysBYX5ZnvZpHUTfG3aU0XE3PQXq3AOfI57QfKQHXGaiyfGytntUQyHxe43j3JJCZhZiNwrkqM0hMva6I+9tEkZ352NRBHciiT+ckF/FFkVzv2qnQVZhBZwHP9q4DFPaZg5QgjXHznTtW5WyjoXU0llA+gyTfn07JN1KTSqlv2CJO1c89l4jgBs9fRUMLL0iAwEXxchj+C/xXFdFetwmsZm7DgA== X-OriginatorOrg: rmanet.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2017 17:56:10.5070 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR02MB2321 archived-at: Wed, 31 May 2017 17:56:24 -0000 I'm using TcpDiscovery with the nodes running at Amazon. So far the networking inside the Amazon cloud has been rock solid. That said, over the weekend I had two compute tasks fail because of an empty projection. It looks like the compute nodes all disconnected. Its unclear why exactly the nodes disconnected. I'm thinking of increasing the number of missed heartbeats but I'd also like to increase the amount of logging so that I have more information to debug similar errors in the future. I know that Ignite logging can be verbose - does anyone have suggestions for how I'd want to configure the logging in order to capture clues to diagnose the random disconnects but also not have a mountain of useless logs? Thanks! Ryan