hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DSuiter RDX <dsui...@rdx.com>
Subject Re: Job initialization failed: java.lang.NullPointerException at resolveAndAddToTopology
Date Fri, 11 Oct 2013 12:19:54 GMT
The user running the job (might not be your username depending on your
setup) does not appear to have executable permissions on the jobtracker
cluster topology python script - I'm basing this on the lines:

2013-10-11 10:24:53,035 WARN org.apache.hadoop.net.ScriptBasedMapping:
Exception running
/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py
10.160.25.249
java.io.IOException: Cannot run program
"/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py" (in
directory "/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER"):
java.io.IOException: error=13, Permission denied

So checking on the permissions for that file, determining what user is
kicking off your job, which depends on how you submit it, and making sure
that user has the execute permission on that file will probably fix this.

If you are using a management console, such as Cloudera SCM, when you
submit jobs, they are run as an application user, so, Flume services run
under the "Flume" user, HBase jobs will typically run under the HBase user,
and so on. It can cause some surprises if you do not expect it.

*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com


On Fri, Oct 11, 2013 at 7:59 AM, fab wol <darkwolli32@gmail.com> wrote:

> Hey everyone, I've got supplied with a decent ten node CDH 4.4 cluster,
> only 7 days old, and someone tried some HBase stuff on it. Now I wanted to
> try some MR Stuff on it, but starting a Job is already not possible (even
> the wordcount example). The error log of the jobtracker produces a log 700k
> lines long but it consists mainly of these lines repeatedly:
>
> 2013-10-11 10:24:53,033 INFO org.apache.hadoop.mapred.JobTracker: Lost
> tracker 'tracker_z-asanode02:localhost/127.0.0.1:53712'
> 2013-10-11 10:24:53,033 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:mapred (auth:SIMPLE) cause:java.io.IOException:
> java.lang.NullPointerException
> 2013-10-11 10:24:53,034 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 22 on 8021, call
> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@13b31acd, true,
> true, true, -1), rpc version=2, client version=32,
> methodsFingerPrint=-159967141 from 10.160.25.250:44389: error:
> java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2751)
>  at
> org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2731)
>  at
> org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3227)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2931)
>  at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1751)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1747)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1745)
> 2013-10-11 10:24:53,035 WARN org.apache.hadoop.net.ScriptBasedMapping:
> Exception running
> /run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py
> 10.160.25.249
> java.io.IOException: Cannot run program
> "/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER/topology.py" (in
> directory "/run/cloudera-scm-agent/process/556-mapreduce-JOBTRACKER"):
> java.io.IOException: error=13, Permission denied
>  at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>  at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
> at org.apache.hadoop.util.Shell.run(Shell.java:188)
>  at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>  at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
>  at
> org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
> at
> org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
>  at
> org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2750)
>  at
> org.apache.hadoop.mapred.JobTracker.addNewTracker(JobTracker.java:2731)
> at
> org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3227)
>  at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2931)
>  at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1751)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1747)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1745)
> Caused by: java.io.IOException: java.io.IOException: error=13, Permission
> denied
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>  at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>  ... 21 more
>
> it doesn't matter if it is a pure hadoop job or a oozie submitted job.
> there seems to be something wrong in the basic configuration. Anyone an
> idea?
>
> Cheers
> Wolli
>

Mime
View raw message