hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cosmin Lehene (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-630) In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block.
Date Tue, 22 Dec 2009 15:28:29 GMT

    [ https://issues.apache.org/jira/browse/HDFS-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793640#action_12793640
] 

Cosmin Lehene commented on HDFS-630:
------------------------------------

@stack unfortunately, no. The patch needs to be changed for trunk. 
{code:title=ClientProtocol.java}
Index: src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
===================================================================
--- src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java	(revision 891402)
+++ src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java	(working copy)
@@ -44,9 +44,9 @@
    * Compared to the previous version the following changes have been introduced:
    * (Only the latest change is reflected.
    * The log of historical changes can be retrieved from the svn).
-   * 50: change LocatedBlocks to include last block information.
+   * 51: changed addBlock to include a list of excluded datanodes.
    */
-  public static final long versionID = 50L;
+  public static final long versionID = 51L;
{code}

The versionID in 0.21 changes from 50L to 51L. The problem is that on trunk is already 52L
so it should probably change it from 52L to 53L. This could be, however ignored on trunk and
changed independently. I'm not sure what's the right approach. I could create another patch
for trunk, however this would just poise versionID meaningless - It's 51L on 0.21, but on
trunk 51L is something else. 

> In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes
when locating the next block.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-630
>                 URL: https://issues.apache.org/jira/browse/HDFS-630
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client, name-node
>    Affects Versions: 0.21.0
>            Reporter: Ruyue Ma
>            Assignee: Cosmin Lehene
>         Attachments: 0001-Fix-HDFS-630-0.21-svn.patch, 0001-Fix-HDFS-630-for-0.21-and-trunk-unified.patch,
0001-Fix-HDFS-630-for-0.21.patch, 0001-Fix-HDFS-630-svn.patch, 0001-Fix-HDFS-630-svn.patch,
0001-Fix-HDFS-630-trunk-svn-1.patch, 0001-Fix-HDFS-630-trunk-svn-2.patch, HDFS-630.patch
>
>
> created from hdfs-200.
> If during a write, the dfsclient sees that a block replica location for a newly allocated
block is not-connectable, it re-requests the NN to get a fresh set of replica locations of
the block. It tries this dfs.client.block.write.retries times (default 3), sleeping 6 seconds
between each retry ( see DFSClient.nextBlockOutputStream).
> This setting works well when you have a reasonable size cluster; if u have few datanodes
in the cluster, every retry maybe pick the dead-datanode and the above logic bails out.
> Our solution: when getting block location from namenode, we give nn the excluded datanodes.
The list of dead datanodes is only for one block allocation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message