hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bo Dong (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6121) High Availability support for Hadoop
Date Fri, 03 Jul 2009 08:55:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726835#action_12726835

Bo Dong commented on HADOOP-6121:

To Amr:
We found that Bookkeeper said Hadoop used Write-Ahead logging approach. 
However, based on my understanding, in 0.19, Hadoop modifies the contents in memory first,
and then persists the log. And there is no transaction relationship between modifying memory
and persisting log.
For example, in the Create operation (Create a new file entry in the namespace)
[FSNamesystem.java 998] startFileInternal(src, permissions, holder, clientMachine, overwrite,
false, replication, blockSize); 
// Add a node child to the namespace, and write the log to a buffer.
[FSNamesystem.java 1000] getEditLog().logSync(); 
// persist the log to the disk.

> High Availability support for Hadoop
> ------------------------------------
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as
well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern
to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves
and how commit "write" operation to NameNode after the agreement among primary/standby/slaves
on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help
of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker?
Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also
needs the same approach from NameNode for HA support.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message