hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <ma...@cloudera.com>
Subject Re: fair scheduler making jobs fail?
Date Wed, 02 Dec 2009 03:46:14 GMT
Did you place the fair scheduler on your classpath? Can you see its UI if
you go to http://<jobtracker>:50030/scheduler ?

On Mon, Nov 30, 2009 at 4:04 PM, Mike Kendall <mkendall@justin.tv> wrote:

> startup sequence is fine.  there is no log file generated, just the xml and
> jar.
>
> the jobtracker gives this on failure:
>
> 2009-11-30 15:51:33,314 WARN org.apache.hadoop.mapred.JobInProgress:
> Running cache for maps missing!! Job details are missing.
> 2009-11-30 15:51:33,314 WARN org.apache.hadoop.mapred.JobInProgress:
> Non-running cache for maps missing!! Job details are missing.
> 2009-11-30 15:51:33,314 INFO org.apache.hadoop.mapred.JobTracker:
> Removed completed task 'attempt_200911301535_0003_m_000113_1' from
> 'tracker_hadoop2.justin.tv:localhost/127.0.0.1:43323'
> 2009-11-30 15:52:09,469 INFO org.apache.hadoop.mapred.TaskInProgress:
> Error from attempt_200911301535_0002_r_000000_3:
> java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
> failed with code 1
>        at
> org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:311)
>        at
> org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540)
>        at
> org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:130)
>        at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
>        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> ?_?
>
>
> On Mon, Nov 30, 2009 at 3:55 PM, Todd Lipcon <todd@cloudera.com> wrote:
> > where? copy paste the startup sequence and the job submission logs from
> the
> > JT log?
> >
> > You gotta provide some details here :)
> >
> > -Todd
> >
> > On Mon, Nov 30, 2009 at 3:53 PM, Mike Kendall <mkendall@justin.tv>
> wrote:
> >
> >> java runtime error, exit code 1..
> >>
> >> On Mon, Nov 30, 2009 at 3:52 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >> > Any errors in your jobtracker log? Usually you'll see something there
> if
> >> the
> >> > scheduler fails to start.
> >> >
> >> > What errors are the jobs failing with?
> >> >
> >> > -Todd
> >> >
> >> > On Mon, Nov 30, 2009 at 3:49 PM, Mike Kendall <mkendall@justin.tv>
> >> wrote:
> >> >
> >> >> no dice...  and the default configuration from
> >> >> http://hadoop.apache.org/common/docs/current/fair_scheduler.html
> >> >> didn't work either.
> >> >>
> >> >> On Mon, Nov 30, 2009 at 3:04 PM, Allen Wittenauer
> >> >> <awittenauer@linkedin.com> wrote:
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 11/30/09 2:59 PM, "Mike Kendall" <mkendall@justin.tv>
wrote:
> >> >> >
> >> >> >> so i'm working on a cluster with one other guy and we decided
to
> try
> >> >> >> the fair scheduler but found that it caused all of our jobs
to
> fail.
> >> >> >>
> >> >> >> has anyone else had this issue?  is there something more to
the
> >> >> >> configuration other than:
> >> >> >
> >> >> > You probably need:
> >> >> >
> >> >> > <property>
> >> >> >  <name>mapred.fairscheduler.allocation.file</name>
> >> >> >  <value>/some/path/fairshare-pools.xml</value>
> >> >> > </property>
> >> >> >
> >> >> > ... and then put ...
> >> >> >
> >> >> > <?xml version="1.0"?>
> >> >> > <allocations>
> >> >> > </allocations>
> >> >> >
> >> >> >
> >> >> > in fairshare-pools.xml.
> >> >> >
> >> >> >
> >> >>
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message