hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-4584) Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode
Date Tue, 24 Feb 2009 17:40:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676344#action_12676344
] 

rangadi edited comment on HADOOP-4584 at 2/24/09 9:39 AM:
---------------------------------------------------------------

Ideally there is no requirement for block reports. It is essentially to used as 'catch all'
for various bugs and errors. (of course, it is now overloaded with job of informing about
deletions to NameNode and should be separated).

Yes, it specifically removes disk scan without fundamentally changing meaning of block reports.
Now DN informs NameNode about the the block that it thinks it has. because :

 * 'rm -r' by admin is just one form of many many things that can go wrong with blocks on
datanode. There is no perticular reason we should have this very costly disk scan (with a
global lock held) just for this. 
    ** In fact 'rm -r' is probably the least probable error (haven't seen even once in practice).


 * We have periodic block verification that does handle various things that can go wrong with
a block (it can improve further). 
      ** So 'rm -r' will be handled, just at the rate of rest of the block problems.
 
 * on the other hand many users have complained about datanode scans taking 10s of minutes
and making datanodes lose heartbeats. 
     ** This makes the system pretty unusable and a major obstruction for graceful degradation
under load and for scalability.
    ** One can argue that those users should not have so many blocks. But I think DN should
still handle it to the best of it abilities and not die on them.
    ** Disks might be slow for many other reasons (other tasks on the machine, etc).

  * I think this is orthogonal to HADOOP-1079 since it addresses RPC and NameNode overhead
of block reports. This jira is only about DataNode side.

Yes, this is a bigger change in semantics than what we proposed earlier : to scan the directories
slowly, without holding the global lock... but offline scan looks like a workaround for a
problem that does not need to be solved. Not scanning is much simpler than handling offline
scan.

Eventually we need to reduce the frequency of block reports.. this can be done as soon as
we add acks for block deletions. This JIRA is major step in that direction.

 


      was (Author: rangadi):
    Ideally there is no requirement for block reports. It is essentially to used as 'catch
all' for various bugs and errors. (of course, it is now overloaded with job of informing about
deletions to NameNode, this should removed).

Yes, it specifically removes disk scan without fundamentally changing meaning of block reports.
Now DN informs NameNode about the the block that it thinks it had. because :

 * 'rm -r' by admin is just one form of many many things that can go wrong with blocks on
datanode. There is no perticular reason we should have this very costly disk scan (with a
global lock held) just for this. 
    ** In fact 'rm -r' is probably the least probable error (haven't seen even once in practice).


 * We have periodic block verification that does handle various things that can go wrong with
a block (it can improve further). 
      ** So 'rm -r' will be handled, just at the rate of rest of the block problems.
 
 * on the other hand many users have complained about datanode scans taking 10s of minutes
and making datanodes lose heartbeats. 
     ** This makes the system pretty unusable and a major obstruction for graceful degradation
under load and for scalability.
    ** One can argue that those users should not have so many blocks. But I think DN should
still handle it to the best of it abilities and not die on them.
    ** Disks might be slow for many other reasons (other tasks on the machine, etc).

  * I think this is orthogonal to HADOOP-1079 since it addresses RPC and NameNode overhead
of block reports. This jira is only about DataNode side.

Yes, this is a bigger change in semantics than what we proposed earlier : to scan the directories
slowly, without holding the global lock... but offline scan looks like a workaround for a
problem that does not need to be solved. Not scanning is much simpler than handling offline
scan.

Eventually we need to reduce the frequency of block reports.. this can be done as soon as
we add acks for block deletions. This JIRA is major step in that direction.

 

  
> Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4584
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Hairong Kuang
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.0
>
>         Attachments: 4584.patch, 4584.patch, 4584.patch, 4584.patch, 4584.patch, 4584.patch
>
>
> sometimes due to disk or some other problems, datanode takes minutes or tens of minutes
to generate a block report. It causes the datanode not able to send heartbeat to NameNode
every 3 seconds. In the worst case, it makes NameNode to detect a lost heartbeat and wrongly
decide that the datanode is dead.
> It would be nice to have two threads instead. One thread is for scanning data directories
and generating block report, and executes the requests sent by NameNode; Another thread is
for sending heartbeats, block reports, and picking up the requests from NameNode. By having
these two threads, the sending of heartbeats will not get delayed by any slow block report
or slow execution of NameNode requests.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message