hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Sharma K S <rohithsharm...@huawei.com>
Subject RE: Cleanup activity on YARN containers
Date Tue, 08 Apr 2014 03:25:46 GMT
Hi Kishore,

       Is jobs are submitted through MapReduce or Is it Yarn Application?


1.      For  MapReduce Framwork, framework itself provides facility to clean up per task level.
            Is there any callback kind of facility, in which I can write some code to be executed
on my container at the end of my application or at the end of that particular container execution?
>>>  You can override setup() and cleanup() for doing initialization and cleanup
of your task. This facility is provided by MapReduce framework.

The call flow of task execution is
             The framework first calls setup(org.apache.hadoop.mapreduce.Mapper.Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BMapper.java%E2%98%83Mapper%E2%98%82%E2%98%82setup%E2%98%82org.apache.hadoop.mapreduce.Mapper.Context>,
followed by map(Object, Object, Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BMapper.java%E2%98%83Mapper%E2%98%82%E2%98%82map%E2%98%82Object%E2%98%82Object%E2%98%82Context>
/ reduce(Object, Iterable, Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BReducer.java%E2%98%83Reducer%E2%98%82%E2%98%82reduce%E2%98%82Object%E2%98%82Iterable%E2%98%82Context>
 for each key/value pair. Finally cleanup(Context)<eclipse-javadoc:%E2%98%82=hadoop-mapreduce-client-core/src%5C/main%5C/java%3Corg.apache.hadoop.mapreduce%7BMapper.java%E2%98%83Mapper%E2%98%82%E2%98%82cleanup%E2%98%82Context>
is called.

Note : In clean up, do not hold container for more than "mapreduce.task.timeout". Because,
once map/reduce is completed, progress will not be sent to applicationmaster(ping is not considered
as status update). If your application is taking more than value configured for "mapreduce.task.timeout",
then application master consider this task as timedout. In such case, you need to increase
value for "mapreduce.task.timeout" based on your cleanup time.



2.       For Yarn Application, completed container's list are sent to ApplicationMaster in
heartbeat.  Here you can do clean up activities for containers.

Hope this will help for you.  :)!!


Thanks & Regards
Rohith Sharma K S

From: Krishna Kishore Bonagiri [mailto:write2kishore@gmail.com]
Sent: 07 April 2014 16:41
To: user@hadoop.apache.org
Subject: Cleanup activity on YARN containers

Hi,

  Is there any callback kind of facility, in which I can write some code to be executed on
my container at the end of my application or at the end of that particular container execution?

 I want to do some cleanup activities at the end of my application, and the clean up is not
related to the localized resources that are downloaded from HDFS.

Thanks,
Kishore

Mime
View raw message