hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlos Valiente (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5257) Export namenode/datanode functionality through a pluggable RPC layer
Date Mon, 30 Mar 2009 21:25:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693911#action_12693911
] 

Carlos Valiente commented on HADOOP-5257:
-----------------------------------------

I don't think test failures are related to this patch:

{code}
java.lang.NullPointerException
	at org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2117)
	at org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2153)
	at org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java:2221)
	at org.apache.hadoop.mapred.TestCapacityScheduler$FakeTaskTrackerManager.killJob(TestCapacityScheduler.java:359)
	at org.apache.hadoop.mapred.TestCapacityScheduler.testClusterBlockingForLackOfMemory(TestCapacityScheduler.java:2069)
{code}

and:

{code}
java.lang.NullPointerException
	at org.apache.hadoop.mapred.JobInProgress.terminateJob(JobInProgress.java:2117)
	at org.apache.hadoop.mapred.JobInProgress.terminate(JobInProgress.java:2153)
	at org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java:2221)
	at org.apache.hadoop.mapred.TestCapacityScheduler$FakeTaskTrackerManager.killJob(TestCapacityScheduler.java:359)
	at org.apache.hadoop.mapred.CapacityTaskScheduler.killJobIfInvalidRequirements(CapacityTaskScheduler.java:1431)
	at org.apache.hadoop.mapred.CapacityTaskScheduler.jobAdded(CapacityTaskScheduler.java:1463)
	at org.apache.hadoop.mapred.JobQueuesManager.jobAdded(JobQueuesManager.java:183)
	at org.apache.hadoop.mapred.TestCapacityScheduler$FakeTaskTrackerManager.submitJob(TestCapacityScheduler.java:387)
	at org.apache.hadoop.mapred.TestCapacityScheduler.submitJob(TestCapacityScheduler.java:625)
	at org.apache.hadoop.mapred.TestCapacityScheduler.testHighMemoryJobWithInvalidRequirements(TestCapacityScheduler.java:1992)
{code}

> Export namenode/datanode functionality through a pluggable RPC layer
> --------------------------------------------------------------------
>
>                 Key: HADOOP-5257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5257
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: Carlos Valiente
>            Priority: Minor
>         Attachments: HADOOP-5257-v2.patch, HADOOP-5257-v3.patch, HADOOP-5257-v4.patch,
HADOOP-5257-v5.patch, HADOOP-5257-v6.patch, HADOOP-5257-v7.patch, HADOOP-5257-v8.patch, HADOOP-5257.patch
>
>
> Adding support for pluggable components would allow exporting DFS functionallity using
arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion
in HADOOP-4707.
> Plug-in implementations would extend this base class:
> {code}abstract class Plugin {
>     public abstract datanodeStarted(DataNode datanode);
>     public abstract datanodeStopping();
>     public abstract namenodeStarted(NameNode namenode);
>     public abstract namenodeStopping();
> }{code}
> Name node instances would then start the plug-ins according to a configuration object,
and would also shut them down when the node goes down:
> {code}public class NameNode {
>     // [..]
>     private void initialize(Configuration conf)
>         // [...]
>         for (Plugin p: PluginManager.loadPlugins(conf))
>           p.namenodeStarted(this);
>     }
>     // [..]
>     public void stop() {
>         if (stopRequested)
>             return;
>         stopRequested = true;
>         for (Plugin p: plugins) 
>             p.namenodeStopping();
>         // [..]
>     }
>     // [..]
> }{code}
> Data nodes would do a similar thing in {{DataNode.startDatanode()}} and {{DataNode.shutdown}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message