hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: custom task cleanup even when task is killed?
Date Mon, 13 Sep 2010 20:39:18 GMT
Have you tried this ?
        Runtime.getRuntime().addShutdownHook(new ShutdownThread());


On Mon, Sep 13, 2010 at 1:33 PM, Chase Bradford <chase.bradford@gmail.com>wrote:

> Thanks David,
>
> Unfortunately, that's only called when a task finishes consuming input
> successfully.  My issue deals with tasks that are killed (job is
> killed or task is pre-empted by the scheduler).  I tried overriding
> run() as follows:
>
> public void run(Context context) {
>  try{
>    super.run();
>  finally{
>    // my very important cleanup stuff that should very rarely get missed.
>  }
> }
>
> If the task fails by throwing an unhandled exception, then cleanup()
> is skipped (the default run doesn't call it), but the finally code
> still works.  However, if the task tracker kills the task, then the
> finally block is skipped.  I'm reluctant to put in signal handling
> code to catch the TERM signal, as that's not Java standard.
>
> Thanks Again,
> Chase
>
> On Mon, Sep 13, 2010 at 11:28 AM, David Rosenstrauch <darose@darose.net>
> wrote:
> > On 09/12/2010 02:38 AM, Chase Bradford wrote:
> >>
> >> I have a mapper class (extended from mapreduce.Mapper), where setup
> >> reports to an outside resource.  I want to make sure that most of the
> >> time when the task fails or is killed, a specific chunk of cleanup
> >> code is executed.
> >
> > Just like there's a "setup(Mapper.Context context)" method, there's also
> a
> > "cleanup(Mapper.Context context)" method for just this purpose.
> >
> > See:
> >
> http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/Mapper.html#cleanup(org.apache.hadoop.mapreduce.Mapper.Context)<http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/Mapper.html#cleanup%28org.apache.hadoop.mapreduce.Mapper.Context%29>
> >
> > DR
> >
>
>
>
> --
> Chase Bradford
>
>
> “If in physics there's something you don't understand, you can always
> hide behind the uncharted depths of nature. But if your program
> doesn't work, there is no obstinate nature. If it doesn't work, you've
> messed up.”
>
> - Edsger Dijkstra
>

Mime
View raw message