Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1767907056.1233905639799.JavaMail.jira@brutus>
Date: Thu, 5 Feb 2009 23:33:59 -0800 (PST)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4584) Slow generation of blockReport at
 DataNode causes delay of sending heartbeat to NameNode
In-Reply-To: <178281774.1225759904336.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671031#action_12671031 ] 

Raghu Angadi commented on HADOOP-4584:
--------------------------------------


Synchronization is not correct. 

E.g. : The condition should be checked inside the lock that you wait on. It is not fatal in this case only because it waits only for 1 sec, but still it is better not to write that way. The block report thread should ideally do "{{wait(time till next report)}}", then the bug shows up more.  A correct synchronization should work irrespective of wait time.

Similarly commandQueue and receviedBlockList sizes are checked outside the locks. We don't need to synchronize on these two any more (since the patch uses common "dataAvailable" for synchronization).

I don't think new class for CommandQueue is required. Also no need for conversion between list and arrays etc. I would synchronize the following way :

{code}
// Block receiver thread :

   while (1) {
        cmd = null;
        synchronized (dataAvaliable) {
             cmd = commandQ.dequeue();
             if (cmd == null) {
                break;
         }
         processCmd(cmd);
         // similarly receivedBlockList. RPC should called outside lock.
    }

   // ...
   synchronized (dataAvailable) {
        if (cmdQ.size() == 0 || receivedBlockList.size() == 0) {
           dataAvailable.wait (time till next report);
       }
   }
                     
// heartBeat thread : 
   synchronize (dataAvailable) {
         cmdQ.addAll(cmds);
         dataAvailable.notify();
    }

// writers :
   synchronized (dataAvailable) {
        receivedBlockArray.add(block);
        dataAvailable.notify();
   }

// etc.. may be locking around delHints can also be removed.. though not related here.
    
{code}


> Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4584
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Hairong Kuang
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.0
>
>         Attachments: 4584.patch, 4584.patch, 4584.patch
>
>
> sometimes due to disk or some other problems, datanode takes minutes or tens of minutes to generate a block report. It causes the datanode not able to send heartbeat to NameNode every 3 seconds. In the worst case, it makes NameNode to detect a lost heartbeat and wrongly decide that the datanode is dead.
> It would be nice to have two threads instead. One thread is for scanning data directories and generating block report, and executes the requests sent by NameNode; Another thread is for sending heartbeats, block reports, and picking up the requests from NameNode. By having these two threads, the sending of heartbeats will not get delayed by any slow block report or slow execution of NameNode requests.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.