Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAGUrFMBc42M44zJ0caAHUtzXyxZYwW=MpV8a2+mWqk-guSDUdQ@mail.gmail.com>
References: 
 <CAGUrFMBc42M44zJ0caAHUtzXyxZYwW=MpV8a2+mWqk-guSDUdQ@mail.gmail.com>
From: Josh Wills <josh.wills@gmail.com>
Date: Fri, 30 Oct 2015 08:49:31 -0700
Message-ID: 
 <CANb5z2+tX5XFzwJ+s7oMHakxRfJGo=ueXVVy5xQJj4664WyzDQ@mail.gmail.com>
Subject: Re: NullPointerExceptions in handleMultiPaths CompletionHook
To: dev <dev@crunch.apache.org>
Content-Type: multipart/alternative; boundary=001a113b6976bcda2c05235460bd

--001a113b6976bcda2c05235460bd
Content-Type: text/plain; charset=UTF-8

David! Welcome back!

I haven't hit that one before; if you tweak handleMultiPaths to look like
the below, does it fix the issue?

J

private synchronized void handleMultiPaths(MRJob job) throws IOException {
  try {
    if (job.getJobState() == MRJob.State.SUCCESS) {
      if (!multiPaths.isEmpty()) {
        for (Map.Entry<Integer, PathTarget> entry : multiPaths.entrySet()) {
          entry.getValue().handleOutputs(job.getJob().getConfiguration(),
workingPath, entry.getKey());
        }
      }
    }  } catch(Exception ie) {
    throw new IOException(ie);
  }

}


On Fri, Oct 30, 2015 at 8:21 AM, David Whiting <davidwhiting@gmail.com>
wrote:

> Hi everybody! I'm back and pushing Crunch in a new organisation
>
> I'm having some strange non-deterministic problems with the end of my
> Crunch job executions in a new environment - I've got some possible ideas
> as to why it's happening, but no good ideas for workarounds so I was hoping
> somebody might be able to help me out. Basically, this is what it looks
> like:
>
> 15/10/30 15:01:55 INFO jobcontrol.CrunchControlledJob: Running job
> "crunching.CountEventsByType: SeqFile([{REDACTED}... ID=1 (1/1)"
> 15/10/30 15:01:55 INFO jobcontrol.CrunchControlledJob: Job status available
> at: {REDACTED}/proxy/application_1443106319465_13029/
> 15/10/30 15:05:02 INFO ipc.Client: Retrying connect to server: {REDACTED}.
> Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
> MILLISECONDS)
> 15/10/30 15:05:03 INFO ipc.Client: Retrying connect to server: {REDACTED}.
> Already tried 1 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
> MILLISECONDS)
> 15/10/30 15:05:04 INFO ipc.Client: Retrying connect to server: {REDACTED}.
> Already tried 2 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
> MILLISECONDS)
> 15/10/30 15:05:04 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
> 15/10/30 15:05:04 ERROR exec.MRExecutor: Pipeline failed due to exception
> java.io.IOException: java.lang.NullPointerException
>         at
>
> org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.handleMultiPaths(CrunchJobHooks.java:99)
>         at
>
> org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.run(CrunchJobHooks.java:86)
>         at
>
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.checkRunningState(CrunchControlledJob.java:288)
>         at
>
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.checkState(CrunchControlledJob.java:299)
>         at
>
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.checkRunningJobs(CrunchJobControl.java:201)
>         at
>
> org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:321)
>         at
> org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:131)
>         at
> org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:58)
>         at
> org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:90)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.mapreduce.Job$1.run(Job.java:325)
>         at org.apache.hadoop.mapreduce.Job$1.run(Job.java:322)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
>         at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:632)
>         at
>
> org.apache.crunch.impl.mr.exec.CrunchJobHooks$CompletionHook.handleMultiPaths(CrunchJobHooks.java:91)
>         ... 9 more
>
> The corresponding line in the Hadoop source is this:
>
> return cluster.getClient().getJobStatus(status.getJobID());
>
> The only NPE-generating part of this is that getClient() could return null,
> but I'm not exactly sure what could cause that. We have some intermittent
> problems with our job history server (returning "not found" for whatever
> job it looks up) which could well be correlated to this, but I would expect
> that to fail at the getJobStatus part rather than the getClient part. This
> would, however, agree with the fact the job reports itself as SUCCEEDED
> before it fails during the handleMultiPaths section (as perhaps the request
> to check status there will get routed to the job history server).
>
> This happens with any Crunch jobs I try to run on this cluster, but there
> are plenty of "plain old MapReduce" running on this cluster with no issues,
> so I'm struggling to find reasons why Crunch would fail where the others
> are succeeding.
>
> Thanks,
> David
>

--001a113b6976bcda2c05235460bd--