hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhijit Bagri (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2669) DFS client lost lease during writing into DFS files
Date Sat, 31 May 2008 00:04:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12601309#action_12601309
] 

Abhijit Bagri commented on HADOOP-2669:
---------------------------------------

An OutputStream is removed from prepareCreates when a close() is called on DFSDataOutputStream.
An FSDataOutputStream handle is present with anyone who uses FileSystem.create(). Now if one
calls FSDataOutputStream.close(), it would trigger DFSDataOutputStream.close()(?). Hence,
a FSDataOutputStream.close() call  may potentially bring prepareCreates to zero. 

However, in the meantime a DataStreamer thread may be still writing a block. If this takes
a long time, then at some time it may face lease expiry. Am I missing something?

> DFS client lost lease during writing into DFS files
> ---------------------------------------------------
>
>                 Key: HADOOP-2669
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2669
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Runping Qi
>            Assignee: dhruba borthakur
>         Attachments: NoLease.patch, noLeaseOnFile.patch
>
>
> I have a program that reads a block compressed sequence file, does some processing on
the records and writes the
> processed records into another  block compressed sequence file.
> During execution of the program, I got the following exception: 
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs.LeaseExpiredException: No
lease on xxxxx/part-00000
>         at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:976)
>         at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:293)
>         at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:379)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:596)
>         at org.apache.hadoop.ipc.Client.call(Client.java:482)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
>         at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:1554)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:1500)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:1626)
>         at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:1602)
>         at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:140)
>         at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
>         at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:39)
>         at java.io.DataOutputStream.write(DataOutputStream.java:90)
>         at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.writeBuffer(SequenceFile.java:1181)
>         at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.sync(SequenceFile.java:1198)
>         at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1248)
>         at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:69)
>      

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message