Return-Path: X-Original-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 226A01808E for ; Mon, 19 Oct 2015 01:43:29 +0000 (UTC) Received: (qmail 98297 invoked by uid 500); 19 Oct 2015 01:43:21 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 98203 invoked by uid 500); 19 Oct 2015 01:43:21 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 98182 invoked by uid 99); 19 Oct 2015 01:43:20 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2015 01:43:20 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 41D6CC5590; Mon, 19 Oct 2015 01:43:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.999 X-Spam-Level: ** X-Spam-Status: No, score=2.999 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id NJwztJnY_rRO; Mon, 19 Oct 2015 01:43:18 +0000 (UTC) Received: from mail2.tencent.com (mail2.tencent.com [163.177.67.149]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTP id 046D9205E9; Mon, 19 Oct 2015 01:43:11 +0000 (UTC) Received: from EXHUB-SZMAIL04.tencent.com (unknown [10.14.6.35]) by mail2.tencent.com (Postfix) with ESMTP id 0BBAE17BB4; Mon, 19 Oct 2015 09:43:09 +0800 (CST) Received: from EXMBX-SZMAIL016.tencent.com ([fe80::1c14:d62a:a06f:3866]) by EXHUB-SZMAIL04.tencent.com ([::1]) with mapi id 14.03.0123.003; Mon, 19 Oct 2015 09:43:08 +0800 From: =?gb2312?B?ZGFuaWVkZW5nKLXLt8kp?= To: user , hdfs-dev Subject: =?gb2312?B?16q3ojogREZTQ2xpZW50IGdvdCBkZWFkbG9jayB3aGVuIGNsb3NlIGZpbGUg?= =?gb2312?Q?and_failed_to_renew_lease?= Thread-Topic: DFSClient got deadlock when close file and failed to renew lease Thread-Index: AdEKD1qyLC5AZ6e7RaScGceBGhJQgg== Date: Mon, 19 Oct 2015 01:43:08 +0000 Message-ID: Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [10.13.73.79] Content-Type: multipart/related; boundary="_005_DC1EB8E42E743A4DA247CDF9755B6B923619C817EXMBXSZMAIL016t_"; type="multipart/alternative" MIME-Version: 1.0 --_005_DC1EB8E42E743A4DA247CDF9755B6B923619C817EXMBXSZMAIL016t_ Content-Type: multipart/alternative; boundary="_000_DC1EB8E42E743A4DA247CDF9755B6B923619C817EXMBXSZMAIL016t_" --_000_DC1EB8E42E743A4DA247CDF9755B6B923619C817EXMBXSZMAIL016t_ Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 DQoNCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQpbY2lkOl9Gb3htYWlsLjFAODQz MmFiN2YtMDYwMS1lMjk1LTEzNTMtYjM3MTAzYWY0MGQ5XWRhbmllZGVuZyi1y7fJKQ0KDQq3orz+ yMujuiBkYW5pZWRlbmcotcu3ySk8bWFpbHRvOmRhbmllZGVuZ0B0ZW5jZW50LmNvbT4NCreiy83K sbzko7ogMjAxNS0xMC0xNiAxNTo0NA0KytW8/sjLo7ogaGRmcy1pc3N1ZXM8bWFpbHRvOmhkZnMt aXNzdWVzQGhhZG9vcC5hcGFjaGUub3JnPjsgdXNlckBoYWRvb3AuYXBhY2hlLm9yZzxtYWlsdG86 dXNlckBoYWRvb3AuYXBhY2hlLm9yZz4NCtb3zOKjuiBERlNDbGllbnQgZ290IGRlYWRsb2NrIHdo ZW4gY2xvc2UgZmlsZSBhbmQgZmFpbGVkIHRvIHJlbmV3IGxlYXNlDQpIaSxBbGwNCiAgICBXZSBm b3VuZCBhIGRlYWRsb2NrIGF0IG91ciBIQmFzZSgwLjk4KSBjbHVzdGVyKGFuZCB0aGUgSGFkb29w IFZlcnNpb24gaXMgMi4yLjApLGFuZCBpdCBzaG91bGQgYmUgSERGUyBCVUcsYXQgdGhlIHRpbWUg b3VyIG5ldHdvcmsgaXMgbm90IHN0YWJsZS4NCiBiZWxvdyBpcyB0aGUgc3RhY2s6DQoNCioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioNCkZvdW5kIG9uZSBKYXZhLWxldmVsIGRlYWRsb2NrOg0KPT09PT09PT09 PT09PT09PT09PT09PT09PT09PT0NCiJNZW1TdG9yZUZsdXNoZXIuMSI6DQogIHdhaXRpbmcgdG8g bG9jayBtb25pdG9yIDB4MDAwMDdmZjI3Y2ZhNTIxOCAob2JqZWN0IDB4MDAwMDAwMDJmYWU1ZWJl MCwgYSBvcmcuYXBhY2hlLmhhZG9vcC5oZGZzLkxlYXNlUmVuZXdlciksDQogIHdoaWNoIGlzIGhl bGQgYnkgIkxlYXNlUmVuZXdlcjpoYmFzZWFkbWluQGhiYXNlLW5zLWdkdC1zaC1tYXJ2ZWwiDQoi TGVhc2VSZW5ld2VyOmhiYXNlYWRtaW5AaGJhc2UtbnMtZ2R0LXNoLW1hcnZlbCI6DQogIHdhaXRp bmcgdG8gbG9jayBtb25pdG9yIDB4MDAwMDdmZjJlNjdlMTZhOCAob2JqZWN0IDB4MDAwMDAwMDQ4 NmNlNjYyMCwgYSBvcmcuYXBhY2hlLmhhZG9vcC5oZGZzLkRGU091dHB1dFN0cmVhbSksDQogIHdo aWNoIGlzIGhlbGQgYnkgIk1lbVN0b3JlRmx1c2hlci4wIg0KIk1lbVN0b3JlRmx1c2hlci4wIjoN CiAgd2FpdGluZyB0byBsb2NrIG1vbml0b3IgMHgwMDAwN2ZmMjdjZmE1MjE4IChvYmplY3QgMHgw MDAwMDAwMmZhZTVlYmUwLCBhIG9yZy5hcGFjaGUuaGFkb29wLmhkZnMuTGVhc2VSZW5ld2VyKSwN CiAgd2hpY2ggaXMgaGVsZCBieSAiTGVhc2VSZW5ld2VyOmhiYXNlYWRtaW5AaGJhc2UtbnMtZ2R0 LXNoLW1hcnZlbCINCg0KSmF2YSBzdGFjayBpbmZvcm1hdGlvbiBmb3IgdGhlIHRocmVhZHMgbGlz dGVkIGFib3ZlOg0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09DQoiTWVtU3RvcmVGbHVzaGVyLjEiOg0KYXQgb3JnLmFwYWNoZS5oYWRvb3AuaGRmcy5M ZWFzZVJlbmV3ZXIuYWRkQ2xpZW50KExlYXNlUmVuZXdlci5qYXZhOjIxNikNCi0gd2FpdGluZyB0 byBsb2NrIDwweDAwMDAwMDAyZmFlNWViZTA+IChhIG9yZy5hcGFjaGUuaGFkb29wLmhkZnMuTGVh c2VSZW5ld2VyKQ0KYXQgb3JnLmFwYWNoZS5oYWRvb3AuaGRmcy5MZWFzZVJlbmV3ZXIuZ2V0SW5z dGFuY2UoTGVhc2VSZW5ld2VyLmphdmE6ODEpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oZGZzLkRG U0NsaWVudC5nZXRMZWFzZVJlbmV3ZXIoREZTQ2xpZW50LmphdmE6NjQ4KQ0KYXQgb3JnLmFwYWNo ZS5oYWRvb3AuaGRmcy5ERlNDbGllbnQuZW5kRmlsZUxlYXNlKERGU0NsaWVudC5qYXZhOjY1OSkN CmF0IG9yZy5hcGFjaGUuaGFkb29wLmhkZnMuREZTT3V0cHV0U3RyZWFtLmNsb3NlKERGU091dHB1 dFN0cmVhbS5qYXZhOjE4ODIpDQotIGxvY2tlZCA8MHgwMDAwMDAwNTViNjA2Y2IwPiAoYSBvcmcu YXBhY2hlLmhhZG9vcC5oZGZzLkRGU091dHB1dFN0cmVhbSkNCmF0IG9yZy5hcGFjaGUuaGFkb29w LmZzLkZTRGF0YU91dHB1dFN0cmVhbSRQb3NpdGlvbkNhY2hlLmNsb3NlKEZTRGF0YU91dHB1dFN0 cmVhbS5qYXZhOjcxKQ0KYXQgb3JnLmFwYWNoZS5oYWRvb3AuZnMuRlNEYXRhT3V0cHV0U3RyZWFt LmNsb3NlKEZTRGF0YU91dHB1dFN0cmVhbS5qYXZhOjEwNCkNCmF0IG9yZy5hcGFjaGUuaGFkb29w LmhiYXNlLmlvLmhmaWxlLkFic3RyYWN0SEZpbGVXcml0ZXIuZmluaXNoQ2xvc2UoQWJzdHJhY3RI RmlsZVdyaXRlci5qYXZhOjI1MCkNCmF0IG9yZy5hcGFjaGUuaGFkb29wLmhiYXNlLmlvLmhmaWxl LkhGaWxlV3JpdGVyVjIuY2xvc2UoSEZpbGVXcml0ZXJWMi5qYXZhOjQwMikNCmF0IG9yZy5hcGFj aGUuaGFkb29wLmhiYXNlLnJlZ2lvbnNlcnZlci5TdG9yZUZpbGUkV3JpdGVyLmNsb3NlKFN0b3Jl RmlsZS5qYXZhOjk3NCkNCmF0IG9yZy5hcGFjaGUuaGFkb29wLmhiYXNlLnJlZ2lvbnNlcnZlci5T dG9yZUZsdXNoZXIuZmluYWxpemVXcml0ZXIoU3RvcmVGbHVzaGVyLmphdmE6NzgpDQphdCBvcmcu YXBhY2hlLmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIuRGVmYXVsdFN0b3JlRmx1c2hlci5mbHVz aFNuYXBzaG90KERlZmF1bHRTdG9yZUZsdXNoZXIuamF2YTo3NSkNCi0gbG9ja2VkIDwweDAwMDAw MDA1OTg2OWVlZDg+IChhIGphdmEubGFuZy5PYmplY3QpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5o YmFzZS5yZWdpb25zZXJ2ZXIuSFN0b3JlLmZsdXNoQ2FjaGUoSFN0b3JlLmphdmE6ODEyKQ0KYXQg b3JnLmFwYWNoZS5oYWRvb3AuaGJhc2UucmVnaW9uc2VydmVyLkhTdG9yZSRTdG9yZUZsdXNoZXJJ bXBsLmZsdXNoQ2FjaGUoSFN0b3JlLmphdmE6MTk3NCkNCmF0IG9yZy5hcGFjaGUuaGFkb29wLmhi YXNlLnJlZ2lvbnNlcnZlci5IUmVnaW9uLmludGVybmFsRmx1c2hjYWNoZShIUmVnaW9uLmphdmE6 MTc5NSkNCmF0IG9yZy5hcGFjaGUuaGFkb29wLmhiYXNlLnJlZ2lvbnNlcnZlci5IUmVnaW9uLmlu dGVybmFsRmx1c2hjYWNoZShIUmVnaW9uLmphdmE6MTY3OCkNCmF0IG9yZy5hcGFjaGUuaGFkb29w LmhiYXNlLnJlZ2lvbnNlcnZlci5IUmVnaW9uLmZsdXNoY2FjaGUoSFJlZ2lvbi5qYXZhOjE1OTEp DQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIuTWVtU3RvcmVGbHVzaGVy LmZsdXNoUmVnaW9uKE1lbVN0b3JlRmx1c2hlci5qYXZhOjQ3MikNCmF0IG9yZy5hcGFjaGUuaGFk b29wLmhiYXNlLnJlZ2lvbnNlcnZlci5NZW1TdG9yZUZsdXNoZXIuZmx1c2hPbmVGb3JHbG9iYWxQ cmVzc3VyZShNZW1TdG9yZUZsdXNoZXIuamF2YToyMTEpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5o YmFzZS5yZWdpb25zZXJ2ZXIuTWVtU3RvcmVGbHVzaGVyLmFjY2VzcyQ1MDAoTWVtU3RvcmVGbHVz aGVyLmphdmE6NjYpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIuTWVt U3RvcmVGbHVzaGVyJEZsdXNoSGFuZGxlci5ydW4oTWVtU3RvcmVGbHVzaGVyLmphdmE6MjM4KQ0K YXQgamF2YS5sYW5nLlRocmVhZC5ydW4oVGhyZWFkLmphdmE6NzQ0KQ0KIkxlYXNlUmVuZXdlcjpo YmFzZWFkbWluQGhiYXNlLW5zLWdkdC1zaC1tYXJ2ZWwiOg0KYXQgb3JnLmFwYWNoZS5oYWRvb3Au aGRmcy5ERlNPdXRwdXRTdHJlYW0uYWJvcnQoREZTT3V0cHV0U3RyZWFtLmphdmE6MTgyMikNCi0g d2FpdGluZyB0byBsb2NrIDwweDAwMDAwMDA0ODZjZTY2MjA+IChhIG9yZy5hcGFjaGUuaGFkb29w LmhkZnMuREZTT3V0cHV0U3RyZWFtKQ0KYXQgb3JnLmFwYWNoZS5oYWRvb3AuaGRmcy5ERlNDbGll bnQuY2xvc2VBbGxGaWxlc0JlaW5nV3JpdHRlbihERlNDbGllbnQuamF2YTo3ODApDQphdCBvcmcu YXBhY2hlLmhhZG9vcC5oZGZzLkRGU0NsaWVudC5hYm9ydChERlNDbGllbnQuamF2YTo3NTMpDQph dCBvcmcuYXBhY2hlLmhhZG9vcC5oZGZzLkxlYXNlUmVuZXdlci5ydW4oTGVhc2VSZW5ld2VyLmph dmE6NDUzKQ0KLSBsb2NrZWQgPDB4MDAwMDAwMDJmYWU1ZWJlMD4gKGEgb3JnLmFwYWNoZS5oYWRv b3AuaGRmcy5MZWFzZVJlbmV3ZXIpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oZGZzLkxlYXNlUmVu ZXdlci5hY2Nlc3MkNzAwKExlYXNlUmVuZXdlci5qYXZhOjcxKQ0KYXQgb3JnLmFwYWNoZS5oYWRv b3AuaGRmcy5MZWFzZVJlbmV3ZXIkMS5ydW4oTGVhc2VSZW5ld2VyLmphdmE6Mjk4KQ0KYXQgamF2 YS5sYW5nLlRocmVhZC5ydW4oVGhyZWFkLmphdmE6NzQ0KQ0KIk1lbVN0b3JlRmx1c2hlci4wIjoN CmF0IG9yZy5hcGFjaGUuaGFkb29wLmhkZnMuTGVhc2VSZW5ld2VyLmFkZENsaWVudChMZWFzZVJl bmV3ZXIuamF2YToyMTYpDQotIHdhaXRpbmcgdG8gbG9jayA8MHgwMDAwMDAwMmZhZTVlYmUwPiAo YSBvcmcuYXBhY2hlLmhhZG9vcC5oZGZzLkxlYXNlUmVuZXdlcikNCmF0IG9yZy5hcGFjaGUuaGFk b29wLmhkZnMuTGVhc2VSZW5ld2VyLmdldEluc3RhbmNlKExlYXNlUmVuZXdlci5qYXZhOjgxKQ0K YXQgb3JnLmFwYWNoZS5oYWRvb3AuaGRmcy5ERlNDbGllbnQuZ2V0TGVhc2VSZW5ld2VyKERGU0Ns aWVudC5qYXZhOjY0OCkNCmF0IG9yZy5hcGFjaGUuaGFkb29wLmhkZnMuREZTQ2xpZW50LmVuZEZp bGVMZWFzZShERlNDbGllbnQuamF2YTo2NTkpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oZGZzLkRG U091dHB1dFN0cmVhbS5jbG9zZShERlNPdXRwdXRTdHJlYW0uamF2YToxODgyKQ0KLSBsb2NrZWQg PDB4MDAwMDAwMDQ4NmNlNjYyMD4gKGEgb3JnLmFwYWNoZS5oYWRvb3AuaGRmcy5ERlNPdXRwdXRT dHJlYW0pDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5mcy5GU0RhdGFPdXRwdXRTdHJlYW0kUG9zaXRp b25DYWNoZS5jbG9zZShGU0RhdGFPdXRwdXRTdHJlYW0uamF2YTo3MSkNCmF0IG9yZy5hcGFjaGUu aGFkb29wLmZzLkZTRGF0YU91dHB1dFN0cmVhbS5jbG9zZShGU0RhdGFPdXRwdXRTdHJlYW0uamF2 YToxMDQpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oYmFzZS5pby5oZmlsZS5BYnN0cmFjdEhGaWxl V3JpdGVyLmZpbmlzaENsb3NlKEFic3RyYWN0SEZpbGVXcml0ZXIuamF2YToyNTApDQphdCBvcmcu YXBhY2hlLmhhZG9vcC5oYmFzZS5pby5oZmlsZS5IRmlsZVdyaXRlclYyLmNsb3NlKEhGaWxlV3Jp dGVyVjIuamF2YTo0MDIpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIu U3RvcmVGaWxlJFdyaXRlci5jbG9zZShTdG9yZUZpbGUuamF2YTo5NzQpDQphdCBvcmcuYXBhY2hl LmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIuU3RvcmVGbHVzaGVyLmZpbmFsaXplV3JpdGVyKFN0 b3JlRmx1c2hlci5qYXZhOjc4KQ0KYXQgb3JnLmFwYWNoZS5oYWRvb3AuaGJhc2UucmVnaW9uc2Vy dmVyLkRlZmF1bHRTdG9yZUZsdXNoZXIuZmx1c2hTbmFwc2hvdChEZWZhdWx0U3RvcmVGbHVzaGVy LmphdmE6NzUpDQotIGxvY2tlZCA8MHgwMDAwMDAwNDg4OGY2ODQ4PiAoYSBqYXZhLmxhbmcuT2Jq ZWN0KQ0KYXQgb3JnLmFwYWNoZS5oYWRvb3AuaGJhc2UucmVnaW9uc2VydmVyLkhTdG9yZS5mbHVz aENhY2hlKEhTdG9yZS5qYXZhOjgxMikNCmF0IG9yZy5hcGFjaGUuaGFkb29wLmhiYXNlLnJlZ2lv bnNlcnZlci5IU3RvcmUkU3RvcmVGbHVzaGVySW1wbC5mbHVzaENhY2hlKEhTdG9yZS5qYXZhOjE5 NzQpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIuSFJlZ2lvbi5pbnRl cm5hbEZsdXNoY2FjaGUoSFJlZ2lvbi5qYXZhOjE3OTUpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5o YmFzZS5yZWdpb25zZXJ2ZXIuSFJlZ2lvbi5pbnRlcm5hbEZsdXNoY2FjaGUoSFJlZ2lvbi5qYXZh OjE2NzgpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIuSFJlZ2lvbi5m bHVzaGNhY2hlKEhSZWdpb24uamF2YToxNTkxKQ0KYXQgb3JnLmFwYWNoZS5oYWRvb3AuaGJhc2Uu cmVnaW9uc2VydmVyLk1lbVN0b3JlRmx1c2hlci5mbHVzaFJlZ2lvbihNZW1TdG9yZUZsdXNoZXIu amF2YTo0NzIpDQphdCBvcmcuYXBhY2hlLmhhZG9vcC5oYmFzZS5yZWdpb25zZXJ2ZXIuTWVtU3Rv cmVGbHVzaGVyLmZsdXNoUmVnaW9uKE1lbVN0b3JlRmx1c2hlci5qYXZhOjQzNSkNCmF0IG9yZy5h cGFjaGUuaGFkb29wLmhiYXNlLnJlZ2lvbnNlcnZlci5NZW1TdG9yZUZsdXNoZXIuYWNjZXNzJDgw MChNZW1TdG9yZUZsdXNoZXIuamF2YTo2NikNCmF0IG9yZy5hcGFjaGUuaGFkb29wLmhiYXNlLnJl Z2lvbnNlcnZlci5NZW1TdG9yZUZsdXNoZXIkRmx1c2hIYW5kbGVyLnJ1bihNZW1TdG9yZUZsdXNo ZXIuamF2YToyNTMpDQphdCBqYXZhLmxhbmcuVGhyZWFkLnJ1bihUaHJlYWQuamF2YTo3NDQpDQoN CkZvdW5kIDEgZGVhZGxvY2suDQoNCioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCg0KDQp0aGUgdGhyZWFkICJNZW1T dG9yZUZsdXNoZXIuMCIgaXMgY2xvc2luZyBvdXRwdXRTdHJlYW0gYW5kIHJlbW92ZSBpdCdzIGxl YXNlIDsNCm90aGVyIHNpZGUgdGhlIGRhZW1vbiB0aHJlYWQgIkxlYXNlUmVuZXdlciIgZmFpbGVk IHRvIGNvbm5lY3QgYWN0aXZlIG5uICBmb3IgcmVuZXdpbmcgIGxlYXNlLGJ1dCAgZ290IFNvY2tl dFRpbWVvdXRFeGNlcHRpb24gICBjYXVzZSBvZiBuZXR3b3JrIGlzIG5vdCBnb29kLHNvIGFib3J0 IG91dHB1dHN0cmVhbS4NCnRoZW4gZGVhZGxvY2sgaXMgbWFkZS4NCg0KYW5kIGl0IHNlZW1zIG5v dCBzb2x2ZWQgYXQgSGFkb29wIDIuNy4xIC5JZiBjb25maXJtZWQgLCB3ZSBjYW4gZml4ZWQgdGhl IGlzc3VlLg0KDQoNCltjaWQ6X0ZveG1haWwuMUA4MTAzMTZkNy1mMTdjLWU5NjgtNDJkZS1lZDAz NjdkYjE2MjRdDQpkYW5pZWRlbmcotcu3ySkNCg0KDQo= --_000_DC1EB8E42E743A4DA247CDF9755B6B923619C817EXMBXSZMAIL016t_ Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable


daniedeng(=B5=CB=B7=C9)
 
=B7=A2=BC=FE=C8=CB=A3=BA daniedeng(=B5=CB=B7=C9)
=B7=A2=CB=CD=CA=B1=BC=E4=A3=BA 2015-10-16 15:44
=CA=D5=BC=FE=C8=CB=A3=BA hdfs-issues; user@hadoop.apache.org
=D6=F7=CC=E2=A3=BA DFSClient got deadlock when close file = and failed to renew lease
Hi,All
    We found a deadlock at our HBase(0.98) cluster(and the Hadoop= Version is 2.2.0),and it should be HDFS BUG,at the time our network is not= stable.
 below is the stack:

***************************************************************************= **********************************************************
Found one Java-level deadlock:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D
"MemStoreFlusher.1":
  waiting to lock monitor 0x00007ff27cfa5218 (object 0x00000002fae5ebe= 0, a org.apache.hadoop.hdfs.LeaseRenewer),
  which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marve= l"
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
  waiting to lock monitor 0x00007ff2e67e16a8 (object 0x0000000486ce662= 0, a org.apache.hadoop.hdfs.DFSOutputStream),
  which is held by "MemStoreFlusher.0"
"MemStoreFlusher.0":
  waiting to lock monitor 0x00007ff27cfa5218 (object 0x00000002fae5ebe= 0, a org.apache.hadoop.hdfs.LeaseRenewer),
  which is held by "LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marve= l"

Java stack information for the threads listed above:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D
"MemStoreFlusher.1":
at org.ap= ache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
- waiting= to lock <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)=
at org.ap= ache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
at org.ap= ache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
at org.ap= ache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
at org.ap= ache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
- locked = <0x000000055b606cb0> (a org.apache.hadoop.hdfs.DFSOutputStream)
at org.ap= ache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.ja= va:71)
at org.ap= ache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
at org.ap= ache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWri= ter.java:250)
at org.ap= ache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
at org.ap= ache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
at org.ap= ache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.jav= a:78)
at org.ap= ache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultSto= reFlusher.java:75)
- locked = <0x000000059869eed8> (a java.lang.Object)
at org.ap= ache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
at org.ap= ache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.ja= va:1974)
at org.ap= ache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795= )
at org.ap= ache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678= )
at org.ap= ache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.= java:472)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(Me= mStoreFlusher.java:211)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.j= ava:66)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlu= sher.java:238)
at java.l= ang.Thread.run(Thread.java:744)
"LeaseRenewer:hbaseadmin@hbase-ns-gdt-sh-marvel":
at org.ap= ache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:1822)
- waiting= to lock <0x0000000486ce6620> (a org.apache.hadoop.hdfs.DFSOutputStre= am)
at org.ap= ache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:780)
at org.ap= ache.hadoop.hdfs.DFSClient.abort(DFSClient.java:753)
at org.ap= ache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:453)
- locked = <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)
at org.ap= ache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.ap= ache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.l= ang.Thread.run(Thread.java:744)
"MemStoreFlusher.0":
at org.ap= ache.hadoop.hdfs.LeaseRenewer.addClient(LeaseRenewer.java:216)
- waiting= to lock <0x00000002fae5ebe0> (a org.apache.hadoop.hdfs.LeaseRenewer)=
at org.ap= ache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:81)
at org.ap= ache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:648)
at org.ap= ache.hadoop.hdfs.DFSClient.endFileLease(DFSClient.java:659)
at org.ap= ache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1882)
- locked = <0x0000000486ce6620> (a org.apache.hadoop.hdfs.DFSOutputStream)
at org.ap= ache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.ja= va:71)
at org.ap= ache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)
at org.ap= ache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWri= ter.java:250)
at org.ap= ache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:402)
at org.ap= ache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:974)
at org.ap= ache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.jav= a:78)
at org.ap= ache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultSto= reFlusher.java:75)
- locked = <0x00000004888f6848> (a java.lang.Object)
at org.ap= ache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:812)
at org.ap= ache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.ja= va:1974)
at org.ap= ache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1795= )
at org.ap= ache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1678= )
at org.ap= ache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1591)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.= java:472)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.= java:435)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.j= ava:66)
at org.ap= ache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlu= sher.java:253)
at java.l= ang.Thread.run(Thread.java:744)

Found 1 deadlock. 

**********************************************************************


the thread "MemStoreFlusher.0&qu= ot; is closing outputStream and remove it's lease ;
other side the daemon thread "LeaseRenewer" failed to connect active nn  for renewing  lease,but  got S= ocketTimeoutException   cause of network is not good,so abort outputst= ream.
then deadlock is made.

and it seems not solved at Hadoop 2.7.1 .If confirmed , we can fixed the= issue.


daniedeng(=B5=CB=B7=C9)


--_000_DC1EB8E42E743A4DA247CDF9755B6B923619C817EXMBXSZMAIL016t_-- --_005_DC1EB8E42E743A4DA247CDF9755B6B923619C817EXMBXSZMAIL016t_--