hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: Cleanup activity on YARN containers
Date Wed, 09 Apr 2014 05:47:45 GMT
  Is there something like shutdown hook for containers?
>> There is no containers specific shutdown hook.

I was telling about Java shutdown hook i.e 'Runtime.getRuntime().addShutdownHook(Thread<http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html>
hook)' during start of container JVM. In hook, clean up can be done.

Thanks & Regards
Rohith Sharma K S

From: Krishna Kishore Bonagiri [mailto:write2kishore@gmail.com]
Sent: 09 April 2014 10:49
To: user@hadoop.apache.org
Subject: Re: Cleanup activity on YARN containers

Hi Rohith,

  Is there something like shutdown hook for containers? Can you please also tell me how to
use that?

Thanks,
Kishore

On Wed, Apr 9, 2014 at 8:34 AM, Rohith Sharma K S <rohithsharmaks@huawei.com<mailto:rohithsharmaks@huawei.com>>
wrote:
For local container clean up, can be cleaned at ShutDownHook. !!??

Thanks & Regards
Rohith Sharma K S

From: Krishna Kishore Bonagiri [mailto:write2kishore@gmail.com<mailto:write2kishore@gmail.com>]
Sent: 08 April 2014 20:01
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Cleanup activity on YARN containers

Hi Rohith,

   Thanks for the reply.

  Mine is a YARN application. I have some files that are local to where the containers run
on, and I want to clean them up at the end of the container execution. So, I want to do this
cleanup on the same node my container ran on. With what you are suggesting, I can't delete
the files local to the container.

   Is there any other way?

Thanks,
Kishore

On Tue, Apr 8, 2014 at 8:55 AM, Rohith Sharma K S <rohithsharmaks@huawei.com<mailto:rohithsharmaks@huawei.com>>
wrote:
Hi Kishore,

       Is jobs are submitted through MapReduce or Is it Yarn Application?


1.      For  MapReduce Framwork, framework itself provides facility to clean up per task level.
            Is there any callback kind of facility, in which I can write some code to be executed
on my container at the end of my application or at the end of that particular container execution?
>>>  You can override setup() and cleanup() for doing initialization and cleanup
of your task. This facility is provided by MapReduce framework.

The call flow of task execution is
             The framework first calls setup(org.apache.hadoop.mapreduce.Mapper.Context),
followed by map(Object, Object, Context) / reduce(Object, Iterable, Context)  for each key/value
pair. Finally cleanup(Context) is called.

Note : In clean up, do not hold container for more than "mapreduce.task.timeout". Because,
once map/reduce is completed, progress will not be sent to applicationmaster(ping is not considered
as status update). If your application is taking more than value configured for "mapreduce.task.timeout",
then application master consider this task as timedout. In such case, you need to increase
value for "mapreduce.task.timeout" based on your cleanup time.



2.       For Yarn Application, completed container's list are sent to ApplicationMaster in
heartbeat.  Here you can do clean up activities for containers.

Hope this will help for you.  :)!!


Thanks & Regards
Rohith Sharma K S

From: Krishna Kishore Bonagiri [mailto:write2kishore@gmail.com<mailto:write2kishore@gmail.com>]
Sent: 07 April 2014 16:41
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Cleanup activity on YARN containers

Hi,

  Is there any callback kind of facility, in which I can write some code to be executed on
my container at the end of my application or at the end of that particular container execution?

 I want to do some cleanup activities at the end of my application, and the clean up is not
related to the localized resources that are downloaded from HDFS.

Thanks,
Kishore



Mime
View raw message