Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5DCF5200BE4 for ; Wed, 21 Dec 2016 12:38:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 5C68F160B26; Wed, 21 Dec 2016 11:38:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8118D160B0C for ; Wed, 21 Dec 2016 12:38:06 +0100 (CET) Received: (qmail 67945 invoked by uid 500); 21 Dec 2016 11:38:05 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 67928 invoked by uid 99); 21 Dec 2016 11:38:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Dec 2016 11:38:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 952001A9F30 for ; Wed, 21 Dec 2016 11:38:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.179 X-Spam-Level: ** X-Spam-Status: No, score=2.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=outlook.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id R_7xIteovDGO for ; Wed, 21 Dec 2016 11:38:02 +0000 (UTC) Received: from BLU004-OMC4S15.hotmail.com (blu004-omc4s15.hotmail.com [65.55.111.154]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 562B75F1C2 for ; Wed, 21 Dec 2016 11:38:01 +0000 (UTC) Received: from EUR03-AM5-obe.outbound.protection.outlook.com ([65.55.111.136]) by BLU004-OMC4S15.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Wed, 21 Dec 2016 03:37:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=3mvLtbSfhIMEW5bWIFXoVFQriGfaJLfkIRrTJaytSKg=; b=b5Z1RB6+7slugwHypR8RcFK2lD0G9cQ2nxOIwTkfCLWLurdNbm+yqyTDCiDnoEutN/Nrjj72NRyjUmdiOo9/hbPDBPBSkwqPf/LP9UotrsAyAjExishxDD7NRHVoPyvFsqOmb9gwjpwKP9fZ2w8UOPVgjq7T5HtGslZNSChsOAMZDzruHjzi8wishXWWo9+GlDovESUsJzODK/tFrCvHAvFTPtn3uKJJ4p3BpIal7VKOIOOHiQSFgErMjPe4nAc08pxUlA62A2wcXVsScICBn65q+hVLR0zAVj6szusLOAX7eQlhcsfIhJqWBLeb2otkdPGomdjOUjCsqBjYZpoXUA== Received: from VE1EUR03FT021.eop-EUR03.prod.protection.outlook.com (10.152.18.54) by VE1EUR03HT043.eop-EUR03.prod.protection.outlook.com (10.152.19.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.761.6; Wed, 21 Dec 2016 11:37:28 +0000 Received: from AM3PR08MB0278.eurprd08.prod.outlook.com (10.152.18.58) by VE1EUR03FT021.mail.protection.outlook.com (10.152.18.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.789.10 via Frontend Transport; Wed, 21 Dec 2016 11:37:29 +0000 Received: from AM3PR08MB0278.eurprd08.prod.outlook.com ([fe80::6411:17bf:75b0:ecf6]) by AM3PR08MB0278.eurprd08.prod.outlook.com ([fe80::6411:17bf:75b0:ecf6%14]) with mapi id 15.01.0771.020; Wed, 21 Dec 2016 11:37:29 +0000 From: Richard Startin To: "user@hbase.apache.org" Subject: Re: Lease exception Thread-Topic: Lease exception Thread-Index: AQHSW32oYyqUNXu1LEiKEahWsNAmtKESRjuh Date: Wed, 21 Dec 2016 11:37:29 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: hbase.apache.org; dkim=none (message not signed) header.d=none;hbase.apache.org; dmarc=none action=none header.from=outlook.com; x-incomingtopheadermarker: OriginalChecksum:38BDE10C94153CB495D9C190BF34C373A72E97B380B715942DB4C63694D99CA1;UpperCasedChecksum:C30D6FED3F0DF2C2BD7510E119917479EDE442B7D297DCBE03BCF570DBE95C4A;SizeAsReceived:7422;Count:39 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [BFZoRGi1NKL6C++Tw+e8u57i4amjkHDx] x-incomingheadercount: 39 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1;VE1EUR03HT043;5:TN3tx+M7Xg2ENVtYOoHAQD0pglfrIQ5xfmjxfPLUEMP/PGaNIf4++h/ieLhheAbAFb+535vciwXHuiPc/DB27l2tEzPh0pnS5ZbZKaLXzr6jolnYEaPCtwLb8PiqNyZ3EVlF6JIDO3Kb28icvnFsGQ==;24:ncK3qByZMnJhaxCaGWSdQMSowzq0QlX9kzGH6Aaco6b6rOGaH80EryTENhSuvsnj1ZHOwnNrkEWBwJmQvG7rCzHzaO1yH3DYugUfAaIRvcY=;7:R/rtbfftXiIv0O+BffaWb0bhfjshAFr0/3MGgtcvOtomryvjTR7Hdf4IpVen0vv+Sjp5mSgs1jb2eUFKnP3icMbfy09MA9dKTgfWoOU6JR7B3kACrul3vJJkc/Q+31ZnERR0/73FinC5YdW9RWShrlUiF9XTNxzMbvhPNzzVdmpKv4yJQNo46BUePnvkaLwRTyT17FaJiT56M4VMhdMJ/R++e/GNwVW4v49GdF/w5/Okdi1EbDnBkZi8zDXP6v6pM6ULGh5H6XCFGNzdebNTjr8rnKW8MugMIwE98rqiovX5Sfz0pthOXKq9Ymrdq/5ZJjO3ndr7AxplsEFAFShAbvQfsmsaaRY/a2zCSvDsl1qXqYII8dkq+eLfbEAO/6OMr26t0VXh94IpW5Qn4heFDpF6LedRNFODFYyasNyio0fWIZ5ZTzSIvKSJbkZ+mCaj5fPUx3fgHa3nhQcb1bYuLQ== x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(10019020)(98900003);DIR:OUT;SFP:1102;SCL:1;SRVR:VE1EUR03HT043;H:AM3PR08MB0278.eurprd08.prod.outlook.com;FPR:;SPF:None;LANG:en; x-ms-office365-filtering-correlation-id: 95fde89b-18a8-4e9e-17b5-08d42995beb4 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(1601124038)(5061506293)(5061507293)(1603103113)(1601125047);SRVR:VE1EUR03HT043; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015012)(82015046);SRVR:VE1EUR03HT043;BCL:0;PCL:0;RULEID:;SRVR:VE1EUR03HT043; x-forefront-prvs: 01630974C0 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_AM3PR08MB02786F91AFDA476DA041B42FDC930AM3PR08MB0278eurp_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Dec 2016 11:37:29.1461 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1EUR03HT043 X-OriginalArrivalTime: 21 Dec 2016 11:37:33.0240 (UTC) FILETIME=[9E9F1B80:01D25B7E] archived-at: Wed, 21 Dec 2016 11:38:07 -0000 --_000_AM3PR08MB02786F91AFDA476DA041B42FDC930AM3PR08MB0278eurp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable It means your lease on a region server has expired during a call to results= canner.next(). This happens on a slow call to next(). You can either embrac= e it or "fix" it by making sure hbase.rpc.timeout exceeds hbase.regionserve= r.lease.period. https://richardstartin.com On 21 Dec 2016, at 11:30, Rajeshkumar J > wrote: Hi, I have faced below issue in our production cluster org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '166881' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:221) at org.apache.hadoop.hbase.regionserver.Leases.cancelLease(Leases.java:206) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:= 2491) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.cal= lBlockingMethod(ClientProtos.java:32205) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:744) Can any one explain what is lease exception Thanks, Rajeshkumar J --_000_AM3PR08MB02786F91AFDA476DA041B42FDC930AM3PR08MB0278eurp_--