hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-565) Introduce block committing logic during new block allocation and file close.
Date Tue, 01 Sep 2009 00:45:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749714#action_12749714
] 

Hairong Kuang commented on HDFS-565:
------------------------------------

This patch seems to has another bug. When complete a file, the last block should be committed
before progress check, but the penultimate block should not be completed until progress check
is passed. This bug occasionally fails TestAppend2. Here is the failed stack trace:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot complete block: block does
not satisfy minimal replication requirement.
        at org.apache.hadoop.hdfs.server.namenode.BlockManager.completeBlock(BlockManager.java:272)
        at org.apache.hadoop.hdfs.server.namenode.BlockManager.commitLastBlock(BlockManager.java:254)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1256)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1224)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:633)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:516)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)

        at org.apache.hadoop.ipc.Client.call(Client.java:766)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
        at $Proxy1.complete(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at $Proxy1.complete(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.completeFile(DFSClient.java:3393)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3380)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
        at org.apache.hadoop.hdfs.TestFileAppend2$Workload.run(TestFileAppend2.java:286)


> Introduce block committing logic during new block allocation and file close.
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-565
>                 URL: https://issues.apache.org/jira/browse/HDFS-565
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs client, name-node
>    Affects Versions: Append Branch
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: Append Branch
>
>         Attachments: CommitBlock.patch, CommitBlock.patch
>
>
> {{ClientProtocol}} methods {{addBlock()}} and {{complete()}} need to include additional
parameter - a block, which has been successfully written to data-nodes. By sending this block
to the name-node the client confirms the generation stamp of the block and its length. The
block on the name-node changes its state to committed and will become complete as long as
one of the finalized replicas reported by data-nodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message