hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9617) my java client use muti-thread to put a same file to a same hdfs uri, after no lease error,then client OutOfMemoryError
Date Tue, 12 Jan 2016 08:22:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093521#comment-15093521
] 

Kai Zheng commented on HDFS-9617:
---------------------------------

I knew what you did, as I said above, you were using so many threads writing to the same HDFS
file, which is not supported yet, if it's ever going to be supported. Every time there is
only ONE writer to own the file lease and can write to the file. You stacktrace told it clearly:
LeaseExpiredException, No lease. You just made the NN and your client crazy, causing all sorts
of problems, which I don't think you would need the time to investigate.

> my java client use muti-thread to put a same file to a same hdfs uri, after no lease
error,then client OutOfMemoryError
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-9617
>                 URL: https://issues.apache.org/jira/browse/HDFS-9617
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: zuotingbing
>         Attachments: HadoopLoader.java, LoadThread.java, UploadProcess.java
>
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on /Tmp2/43.bmp.tmp (inode 2913263): File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_2084151715_1,
pendingcreates: 250]
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3358)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3160)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3042)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:615)
> 	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:188)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:476)
> 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1653)
> 	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> 	at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
> 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:391)
> 	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> 	at com.sun.proxy.$Proxy15.addBlock(Unknown Source)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1473)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1290)
> 	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:536)
> my java client(JVM -Xmx=2G) :
> jmap TOP15:
> num     #instances         #bytes  class name
> ----------------------------------------------
>    1:         48072     2053976792  [B
>    2:         45852        5987568  <constMethodKlass>
>    3:         45852        5878944  <methodKlass>
>    4:          3363        4193112  <constantPoolKlass>
>    5:          3363        2548168  <instanceKlassKlass>
>    6:          2733        2299008  <constantPoolCacheKlass>
>    7:           533        2191696  [Ljava.nio.ByteBuffer;
>    8:         24733        2026600  [C
>    9:         31287        2002368  org.apache.hadoop.hdfs.DFSOutputStream$Packet
>   10:         31972         767328  java.util.LinkedList$Node
>   11:         22845         548280  java.lang.String
>   12:         20372         488928  java.util.concurrent.atomic.AtomicLong
>   13:          3700         452984  java.lang.Class
>   14:           981         439576  <methodDataKlass>
>   15:          5583         376344  [S



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message