hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1306) DFS Scalability: Reduce the number of getAdditionalBlock RPCs on the namenode
Date Mon, 30 Apr 2007 20:20:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12492775
] 

Raghu Angadi commented on HADOOP-1306:
--------------------------------------


This will be useful. Couple of thoughts:
 
 # requiring abandonBlock() is not necessary. fileComplete() can provide actual number of
blocks used and Namenode deletes the rest. This allows Namnode to provide more blocks at a
time without worrying about extra cost due to abandonBlock().
 # we can make fileCreate() to provide a set of blocks avoiding one more addBlock().




> DFS Scalability: Reduce the number of getAdditionalBlock RPCs on the namenode
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-1306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1306
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: dhruba borthakur
>
> One of the most-frequently-invoked RPCs in the namenode is the addBlock() RPC. The DFSClient
uses this RPC to allocate one more block for a file that it is currently operating upon. The
scalability of the namenode will improve if we can decrease the number of addBlock() RPCs.
One idea that we want to discuss here is to make addBlock() return more than one block. This
proposal came out of a discussion I had with Ben Reed. 
> Let's say that addBlock() returns n blocks for the file. The namenode already tracks
these blocks using the pendingCreates data structure. The client guarantees that these n blocks
will be used in order. The client also guarantees that if it cannot use a block (dues to whatever
reason), it will inform the namenode using the abandonBlock() RPC. These RPCs are already
supported.
> Another possible optimization : since the namenode has to allocate n blocks for a file,
should it use the same set of datanodes for this set of blocks? My proposal is that if n is
a small number (e.g. 3), it is prudent to allocate the same set of datanodes to host all replicas
for this set of blocks. This will reduce the CPU spent in chooseTargets().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message