Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 4A5FC200BA6 for ; Tue, 18 Oct 2016 12:55:48 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 490D7160ADC; Tue, 18 Oct 2016 10:55:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F2775160ACC for ; Tue, 18 Oct 2016 12:55:46 +0200 (CEST) Received: (qmail 61093 invoked by uid 500); 18 Oct 2016 10:55:44 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 61082 invoked by uid 99); 18 Oct 2016 10:55:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Oct 2016 10:55:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 22A9F1A07EE for ; Tue, 18 Oct 2016 10:55:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.541 X-Spam-Level: *** X-Spam-Status: No, score=3.541 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, HTML_OBFUSCATE_10_20=1.162, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 7AMu3JaLTQBC for ; Tue, 18 Oct 2016 10:55:42 +0000 (UTC) Received: from mail-qk0-f178.google.com (mail-qk0-f178.google.com [209.85.220.178]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B2F225F47D for ; Tue, 18 Oct 2016 10:55:41 +0000 (UTC) Received: by mail-qk0-f178.google.com with SMTP id o68so330317545qkf.3 for ; Tue, 18 Oct 2016 03:55:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=sDofkVQtIX/Tc49AM7tDlYSZ8H4AsZy2buhQV0yd3Wc=; b=exCgQJjH1vHN5atD0dGO9qdOk0NYmGfl6Wy8MVHiy+Nwu8e74iV9B9Oze0Bz/ImCSP fc/ngmfcfFUYMZX7Jtk5EyjgZ4QZIv7JxA0tyoDWtIOXKRiqlh3M/If+BSlxyoN/KDZH Sfy4SinTUCp0xLhrQ/VGXqnQ6/panDSYzaeO4Dr5ok761AFIEiOtdnTT42ErEl/Xic/b 9ccAp0MPwlG5dhPe2ikLOt0zLPvnFthGQ+xWCErOI83Py1CrusL0uyXfk5xgvFbFnd9L 9dAXK/Z7th0nLXlbzebNAGCbIva7NVf8DeC9ubPrRNy9Mg4N5Xyzd9C3zrRlw0j8u6Sr 9Dqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=sDofkVQtIX/Tc49AM7tDlYSZ8H4AsZy2buhQV0yd3Wc=; b=lNoifVuvGy3zLTLihhDFpiTvpDVXSI8XZq4bPe1HQI4bA741k5CQDETiGBueVqFAJt QSHOwCfWQ/d1qmlm5uwVHlgm9qcvmlF0vmbmZRyFhfDi14xJcL8jgc+/MkBY2sSCZSEO Q8mzVyZKQ8uxtp52jTswfI/ayguAs4dIG2KsLE9CoZ5bH6nDhAUTGB6+7Yd21EqRGBji UB9WCQgCwxv4QO5jTGnw28qrp3W6HlGXweg5ZSdjp0GkzX0qdxAMbAYFWnPTlIeJgNpF Kvt/Ya7pvhjAYexWtkqOE5WzywTsL7RP6K2LfZ+5TnI6IuK3ZfWHfp5KYdgUK4rFeI6v 8r9A== X-Gm-Message-State: AA6/9RmpJl8AjNJO5WDj2PdqbkNmS0Atl3c+DNVK5jwgVTPUtKItJyqqYdz0WxIVir7X9d5a5lO7fN8J/32BPg== X-Received: by 10.55.182.4 with SMTP id g4mr1715412qkf.120.1476788123192; Tue, 18 Oct 2016 03:55:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.200.42.145 with HTTP; Tue, 18 Oct 2016 03:55:22 -0700 (PDT) In-Reply-To: References: From: Zhang Jianfeng Date: Tue, 18 Oct 2016 18:55:22 +0800 Message-ID: Subject: =?UTF-8?Q?Re=3A_LeaseExpiredException=3A_No_lease_on_=2Fuser=2Fbiadmin?= =?UTF-8?Q?=2Fanalytic=E2=80=90root=2FSX5XPWPPDPQH=2F=2E?= To: Gaurav Kumar Cc: "user.hadoop" , Rakesh Radhakrishnan Content-Type: multipart/alternative; boundary=94eb2c060cf07dd2b5053f218737 archived-at: Tue, 18 Oct 2016 10:55:48 -0000 --94eb2c060cf07dd2b5053f218737 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Gaurav. For my case, I called the HDFS API to write the reducer result into HDFS directly, not using Spark. 2016-10-17 23:24 GMT+08:00 Gaurav Kumar : > Hi, > > Please also check for coalesced RDD. I encountered the same error while > writing a coalesced rdd/df to HDFS. If this is the case, please use > repartition instead. > > Sent from OnePlus 3 > > Thanks & Regards, > Gaurav Kumar > > On Oct 17, 2016 11:22 AM, "Zhang Jianfeng" wrote: > > Thanks Rakesh for your kind help. Actually during the job only one > reducer result file (for example part-r-2) had this error, other reducers > worked well. > > Best Regards, > Jian Feng > > 2016-10-17 11:49 GMT+08:00 Rakesh Radhakrishnan : > >> Hi Jian Feng, >> >> Could you please check your code and see any possibilities of >> simultaneous access to the same file. Mostly this situation happens when >> multiple clients tries to access the same file. >> >> Code Reference:- https://github.com/apache/hadoop/blob/branch-2.2 >> /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/ha >> doop/hdfs/server/namenode/FSNamesystem.java#L2737 >> >> Best Regards, >> Rakesh >> Intel >> >> On Mon, Oct 17, 2016 at 7:16 AM, Zhang Jianfeng >> wrote: >> >>> Hi , >>> >>> I hit an wired error. On our hadoop cluster (2.2.0), occasionally a >>> LeaseExpiredException is thrown. >>> >>> The stacktrace is as below: >>> >>> >>> *org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.na= menode.LeaseExpiredException): >>> No lease on /user/biadmin/analytic=E2=80=90root/SX5XPWPPDPQH/.execution= s/.at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesy= stem.java:2737)* >>> >>> *at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInterna= l(FSNamesystem.java:2801)* >>> >>> *at >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSName= system.java:2783)* >>> >>> *at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.com >>> plete(NameNodeRpcServer.java:611= )* >>> >>> *at >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTrans= latorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:428)* >>> >>> *at >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$Clie= ntNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:5= 9586)* >>> >>> *at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(= ProtobufRpcEngine.java:585)* >>> >>> *at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)* >>> >>> *at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)* >>> >>> *at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)* >>> >>> *at >>> java.security.AccessController.doPrivileged(AccessController.java:310)* >>> >>> *at javax.security.auth.Subject.do >>> As(Subject.java:573)* >>> >>> *at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati= on.java:1502)* >>> >>> *at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)* >>> >>> *at org.apache.hadoop.ipc.Client.call(Client.java:1347)* >>> >>> *at org.apache.hadoop.ipc.Client.call(Client.java:1300)* >>> >>> *at >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngin= e.java:206)* >>> >>> *at $Proxy7.complete(Unknown Source)* >>> >>> *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* >>> >>> *at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja= va:60)* >>> >>> *at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso= rImpl.java:37)* >>> >>> at java.lang.reflect.Method.invoke(Method.java:611) >>> >>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth >>> od(RetryInvocationHandler.java:186) >>> >>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >>> ryInvocationHandler.java:102) >>> >>> at $Proxy7.complete(Unknown Source) >>> >>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran >>> slatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:371) >>> >>> at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutpu >>> tStream.java:1894) >>> >>> at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream >>> .java:1881) >>> >>> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close( >>> FSDataOutputStream.java:71) >>> >>> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputSt >>> ream.java:104) >>> >>> at java.io.FilterOutputStream.close(FilterOutputStream.java:154) >>> >>> Any help will be appreciated! >>> >>> -- >>> Best Regards, >>> Jian Feng >>> >> >> > > > -- > Best Regards, > Jian Feng > > > --=20 Best Regards, Jian Feng --94eb2c060cf07dd2b5053f218737 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Gaurav. For my case, I called the HDFS API to write= the reducer result into HDFS directly, not using Spark.

2016-10-17 23:24 GMT+08:00 Gau= rav Kumar <gauravkumar37@gmail.com>:

Hi,

Please also check for coalesced RDD. I encountered the same = error while writing a coalesced rdd/df to HDFS. If this is the case, please= use repartition instead.

Sent from OnePlus 3

Thanks & Regards,
Gaurav Kumar


On Oct 17, 2016 1= 1:22 AM, "Zhang Jianfeng" <jzhang.ch@gmail.com> wrote:
Thanks= =C2=A0Rakesh for your kind help. Actually du= ring the job only one reducer result file (for example part-r-2) had this e= rror, other reducers worked well.=C2=A0

Best=C2=A0Regards,
Jian Feng

2016-10-1= 7 11:49 GMT+08:00 Rakesh Radhakrishnan <rakeshr@apache.org>= :
Hi= =C2=A0Jian Feng,

Could you please check your code and see any = possibilities of simultaneous access to the same file. Mostly this situatio= n happens when multiple clients tries to access the same file.


Best = Regards,
Rakesh
Intel

On Mon, Oct 17, 2016 at 7:16 AM, Zhang Jianfeng <j= zhang.ch@gmail.com> wrote:
Hi ,

=C2=A0 =C2=A0 I= hit an wired error. On our hadoop cluster (2.2.0), occasionally a LeaseExp= iredException is thrown.

The stacktrace is as belo= w:
=C2=A0 =C2=A0

org= .apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.= namenode.LeaseExpiredException): No lease on /user/biadmin/analytic=E2= =80=90root/SX5XPWPPDPQH/.executions/.at org.apache.hadoop.hdfs.server.= namenode.FSNamesystem.checkLease(FSNamesystem.java:2737)

<= /div>

at org.apache.hadoop.hdfs= .server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.j= ava:2801)

at= org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile= (FSNamesystem.java:2783)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:611)

at org.apache.hadoop.hdfs.protocolPB.ClientNa= menodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProto= colServerSideTranslatorPB.java:428)

at org.apache.hadoop.hdfs.protocol.proto.Clien= tNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMeth= od(ClientNamenodeProtocolProtos.java:59586)

at org.apache.hadoop.ipc.Protobuf= RpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:5= 85)

at org.apache= .hadoop.ipc.RPC$Server.call(RPC.java:928)

at org.apache.hadoop.ipc.Server$Handle= r$1.run(Server.java:2048)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:204= 4)

at java.securi= ty.AccessController.doPrivileged(AccessController.java:310)

at javax.security.auth.Subjec= t.doAs(Subject.java:573)

at org.apache.hadoop.security.UserGroupInformation.do= As(UserGroupInformation.java:1502)

at org.apache.hadoop.ipc.Server$Handler.run(Ser= ver.java:2042)

at= org.apache.hadoop.ipc.Client.call(Client.java:1347)

at org.apache.hadoop.ipc.Client.c<= wbr>all(Client.java:1300)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Pro= tobufRpcEngine.java:206)

at $Proxy7.complete(Unknown Source)

at sun.reflect.NativeMethodAccessorImp= l.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcc= essorImpl.java:60)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe= thodAccessorImpl.java:37)

= at java.lang.reflect.Method.invoke(Method.java:611)

at org.apache.hadoop.io.retry.RetryInvocationHan= dler.invokeMethod(RetryInvocationHandler.java:186)

at org.apache.hadoop.io.retry.RetryInvocatio= nHandler.invoke(RetryInvocationHandler.java:102)

at $Proxy7.complete(Unknown Source)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenod= eProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB= .java:371)

at org.apache.hadoo= p.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:1894)

at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1881)

at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.= close(FSDataOutputStream.java:71)

at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:104)

at java.io.FilterOutputStream.clo<= wbr>se(FilterOutputStream.java:154)

Any help will be appreciated!

--
B= est Regards,
Jian Feng




--
=
Be= st Regards,
Jian Feng




--
=
Best Rega= rds,
Jian Feng
--94eb2c060cf07dd2b5053f218737--