hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-923) DFS Scalability: datanode heartbeat timeouts cause cascading timeouts of other datanodes
Date Fri, 09 Feb 2007 20:03:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12471803
] 

Hairong Kuang commented on HADOOP-923:
--------------------------------------

Two comments:
1. I feel that it is not neccessary to balance # of transfers when the heartbeat thread picks
up the replication work. First the background thread that computes pendingTransfers has already
balanced the load. Second block replication work needs to be done asap to avoid data loss.
Since the datanode has been assinged the block replication work, no other datanode is able
to pick up the work. If the work does not get to send to the datanode in the current heartbeat,
it has to wait for at least another heartbeat interval.

2. The background thread that computes pendindingTransfer scans only 100 datanodes per interation
and then sleep for 3 seconds. I feel that the approach does not scale well. For example, when
a cluster size becomes 2000, a datanode's work gets computed every 2000/100*3=1min if we ignore
the computation overhead, which is far less frequently than what we do now (every 3 seonds).
Another minor flaw is that the thread uses the index to record the next node to be checked.
But if the heartbeat queue gets updated between two consecutive interations, the index may
not point to the right node.

> DFS Scalability: datanode heartbeat timeouts cause cascading timeouts of other datanodes
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-923
>                 URL: https://issues.apache.org/jira/browse/HADOOP-923
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.10.1
>            Reporter: dhruba borthakur
>         Assigned To: dhruba borthakur
>         Attachments: pendingTransferThread.patch
>
>
> The datanode sends a heartbeat to the namenode every 3 seconds. The namenode processes
the heartbeat and sends  a list of block-to-be-replicated and blocks-to-be-deleted as part
of the heartbeat response.
> At times when a couple of datanodes fail, the heartbeat processing on the namenode becomes
pretty heavyweight. It acquires the global FSNamesystem lock, traverses the neededReplication
structure, generates a list of blocks to be replicated and responds to the heartbeat message.
Determining the list of blocks-to-be-replciated is pretty heavyweight, takes plenty of CPU
and blocks processing of other heartbeats because of the global FSNamesystem lock.
> It would improve scalability a lot if heartbeat processing does not require the FSNamesystem
lock. In fact, the pre-existing "heartbeat" lock already exists for this purpose. 
> I propose that the Heartbeat message be separate from the "retrieve blocks-to-replicate
and blocks-to-delete" messages. The datanode can continue to heartbeat once every 3 seconds
while it can afford to "retrieve blocks-to-replicate" at a much coarser interval. Heartbeat
processing on the namenode will be fast because it does not require the global FSNamesystem
lock. Moreover, a datanode failure will not aggrevate the heartbeat processing time on the
namenode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message