hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3276) Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job on cluster allocated by hod.
Date Mon, 21 Apr 2008 06:46:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590853#action_12590853
] 

Vinod Kumar Vavilapalli commented on HADOOP-3276:
-------------------------------------------------

This particular problem occurs because of improper setup of Mapred system directory in DFS.
But, this can be hit in general when either user submits jobs or HOD continues running script
commands, even while JT is still in the state of initialization due to various reasons. HOD
currently puts in its best effort to avoid this - ping Mapred Jetty if it is alive - the last
thing that can be done to check JT status by HOD from outside without any support in Hadoop.
And as the above situation makes it clear, we definitely need support from Hadoop to know
when JT can really accepts jobs. Filed a Jira (HADOOP-3289) to address this.

> Got org.apache.hadoop.mapred.JobTracker$IllegalStateException while running hadoop job
on cluster allocated by hod.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3276
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.17.0
>            Reporter: Karam Singh
>            Priority: Minor
>
> Allocated cluster using hod of 5 nodes . It returned the both Mapred and Hdfs UI addresses.
Also checked out of nodes on 3 taskterakers nodes were using getTaskTrackers() api. 
> Then tried to run wordcount example and got following -:
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.mapred.JobTracker$IllegalStateException:
Job tracker still initializing
>         at org.apache.hadoop.mapred.JobTracker.ensureRunning(JobTracker.java:1722)
>         at org.apache.hadoop.mapred.JobTracker.getNewJobId(JobTracker.java:1730)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>         at org.apache.hadoop.ipc.Client.call(Client.java:557)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>         at $Proxy1.getNewJobId(Unknown Source)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:696)
> (Here dfs was static with permissions on.)
> On further checking there was problem in static dfs setting, but Web UI was getting started
resulting successful allocation, but was not able create mparedsystem directory causing JobTracker$IllegalStateException.
> So, It seems if Web UI up then hod assumes that Mapred JT is running
> (This is rare condition)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message