Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BBCDD10CCE for ; Wed, 9 Apr 2014 05:48:46 +0000 (UTC) Received: (qmail 51328 invoked by uid 500); 9 Apr 2014 05:48:39 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 50796 invoked by uid 500); 9 Apr 2014 05:48:34 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 50788 invoked by uid 99); 9 Apr 2014 05:48:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 05:48:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rohithsharmaks@huawei.com designates 119.145.14.64 as permitted sender) Received: from [119.145.14.64] (HELO szxga01-in.huawei.com) (119.145.14.64) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 05:48:27 +0000 Received: from 172.24.2.119 (EHLO szxeml212-edg.china.huawei.com) ([172.24.2.119]) by szxrg01-dlp.huawei.com (MOS 4.3.7-GA FastPath queued) with ESMTP id BTV63945; Wed, 09 Apr 2014 13:47:48 +0800 (CST) Received: from SZXEML451-HUB.china.huawei.com (10.82.67.194) by szxeml212-edg.china.huawei.com (172.24.2.181) with Microsoft SMTP Server (TLS) id 14.3.158.1; Wed, 9 Apr 2014 13:47:46 +0800 Received: from SZXEML512-MBS.china.huawei.com ([169.254.8.221]) by szxeml451-hub.china.huawei.com ([10.82.67.194]) with mapi id 14.03.0158.001; Wed, 9 Apr 2014 13:47:46 +0800 From: Rohith Sharma K S To: "user@hadoop.apache.org" Subject: RE: Cleanup activity on YARN containers Thread-Topic: Cleanup activity on YARN containers Thread-Index: AQHPUlIYSwYPYbOojkq1q4THYg99lJsHCUSAgAA6PoCAAVaecP//oXSAgACKOkA= Date: Wed, 9 Apr 2014 05:47:45 +0000 Message-ID: <0EE80F6F7A98A64EBD18F2BE839C91156755D36D@szxeml512-mbs.china.huawei.com> References: <0EE80F6F7A98A64EBD18F2BE839C911567556B36@szxeml512-mbs.china.huawei.com> <0EE80F6F7A98A64EBD18F2BE839C9115675580B3@szxeml512-mbs.china.huawei.com> In-Reply-To: Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.18.168.138] Content-Type: multipart/alternative; boundary="_000_0EE80F6F7A98A64EBD18F2BE839C91156755D36Dszxeml512mbschi_" MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Virus-Checked: Checked by ClamAV on apache.org --_000_0EE80F6F7A98A64EBD18F2BE839C91156755D36Dszxeml512mbschi_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Is there something like shutdown hook for containers? >> There is no containers specific shutdown hook. I was telling about Java shutdown hook i.e 'Runtime.getRuntime().addShutdow= nHook(Thread hook)' during start of container JVM. In hook, clean up can be done. Thanks & Regards Rohith Sharma K S From: Krishna Kishore Bonagiri [mailto:write2kishore@gmail.com] Sent: 09 April 2014 10:49 To: user@hadoop.apache.org Subject: Re: Cleanup activity on YARN containers Hi Rohith, Is there something like shutdown hook for containers? Can you please also= tell me how to use that? Thanks, Kishore On Wed, Apr 9, 2014 at 8:34 AM, Rohith Sharma K S > wrote: For local container clean up, can be cleaned at ShutDownHook. !!?? Thanks & Regards Rohith Sharma K S From: Krishna Kishore Bonagiri [mailto:write2kishore@gmail.com] Sent: 08 April 2014 20:01 To: user@hadoop.apache.org Subject: Re: Cleanup activity on YARN containers Hi Rohith, Thanks for the reply. Mine is a YARN application. I have some files that are local to where the= containers run on, and I want to clean them up at the end of the container= execution. So, I want to do this cleanup on the same node my container ran= on. With what you are suggesting, I can't delete the files local to the co= ntainer. Is there any other way? Thanks, Kishore On Tue, Apr 8, 2014 at 8:55 AM, Rohith Sharma K S > wrote: Hi Kishore, Is jobs are submitted through MapReduce or Is it Yarn Application? 1. For MapReduce Framwork, framework itself provides facility to clea= n up per task level. Is there any callback kind of facility, in which I can write so= me code to be executed on my container at the end of my application or at t= he end of that particular container execution? >>> You can override setup() and cleanup() for doing initialization and cl= eanup of your task. This facility is provided by MapReduce framework. The call flow of task execution is The framework first calls setup(org.apache.hadoop.mapreduce.Ma= pper.Context), followed by map(Object, Object, Context) / reduce(Object, It= erable, Context) for each key/value pair. Finally cleanup(Context) is call= ed. Note : In clean up, do not hold container for more than "mapreduce.task.tim= eout". Because, once map/reduce is completed, progress will not be sent to = applicationmaster(ping is not considered as status update). If your applica= tion is taking more than value configured for "mapreduce.task.timeout", the= n application master consider this task as timedout. In such case, you need= to increase value for "mapreduce.task.timeout" based on your cleanup time. 2. For Yarn Application, completed container's list are sent to Appli= cationMaster in heartbeat. Here you can do clean up activities for contain= ers. Hope this will help for you. :)!! Thanks & Regards Rohith Sharma K S From: Krishna Kishore Bonagiri [mailto:write2kishore@gmail.com] Sent: 07 April 2014 16:41 To: user@hadoop.apache.org Subject: Cleanup activity on YARN containers Hi, Is there any callback kind of facility, in which I can write some code to= be executed on my container at the end of my application or at the end of = that particular container execution? I want to do some cleanup activities at the end of my application, and the= clean up is not related to the localized resources that are downloaded fro= m HDFS. Thanks, Kishore --_000_0EE80F6F7A98A64EBD18F2BE839C91156755D36Dszxeml512mbschi_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

  Is there something like shutdown hook for con= tainers?

>> There is no containers specific shutdown ho= ok.  

 

I was telling about Java shutdown hook i.e ‘Ru= ntime.getRuntime().addShutdownHook(= Thread hook)’ during start of container JVM. In hook, clean up can be done.

 <= /p>

Thanks & Regards=

Rohith Sharma K S

 <= /p>

From: Krishna = Kishore Bonagiri [mailto:write2kishore@gmail.com]
Sent: 09 April 2014 10:49
To: user@hadoop.apache.org
Subject: Re: Cleanup activity on YARN containers

 

Hi Rohith,

 

  Is there something like shutdown hook for con= tainers? Can you please also tell me how to use that?

 

Thanks,

Kishore

 

On Wed, Apr 9, 2014 at 8:34 AM, Rohith Sharma K S &l= t;rohithshar= maks@huawei.com> wrote:

For local container clean up, can be cl= eaned at ShutDownHook. !!??

 

Thanks & Regards<= /p>

Rohith Sharma K S

 

From: Krishna Kishore Bonagi= ri [mailto:wri= te2kishore@gmail.com]
Sent: 08 April 2014 20:01
To: user= @hadoop.apache.org
Subject: Re: Cleanup activity on YARN containers

 

Hi Rohith, 

 

   Thanks for the reply.

 

  Mine is a YARN application. I have some files that are loca= l to where the containers run on, and I want to clean them up at the end of= the container execution. So, I want to do this cleanup on the same node my container ran on. With what you are sugge= sting, I can't delete the files local to the container.

 

   Is there any other way?

 

Thanks,

Kishore

 

On Tue, Apr 8, 2014 at 8:55 AM, Rohith Sharma K S <rohithsharmaks@huawei.co= m> wrote:

Hi Kishore,

 

       Is jobs are submitted through= MapReduce or Is it Yarn Application?

 

1.      = For  MapReduce Framwork, framework itself provides facility to clean u= p per task level.

            Is t= here any callback kind of facility, in which I can write some code to be ex= ecuted on my container at the end of my application or at the end of that particular container execution?

>>>  You can override setup() and = cleanup() for doing initialization and cleanup of your task. This facility = is provided by MapReduce framework.

 

The call flow of task execution is

        =      The framework first calls setup(org.apache.hadoop.mapre= duce.Mapper.Context), followed by map(Object, Object, Context)<= /span> / reduce(Object, Iterable, Cont= ext)  for each key/value pair. Finally cleanup(Context) is called.

 

Note : In clean up, do not hold container for mo= re than “mapreduce.task.timeout”. Because, once map/reduce is c= ompleted, progress will not be sent to applicationmaster(ping is not considered as status update). If your application is taking more than valu= e configured for “mapreduce.task.timeout”, then application mas= ter consider this task as timedout. In such case, you need to increase valu= e for “mapreduce.task.timeout” based on your cleanup time.

 

    

2.       For Yarn Application, completed container= ’s list are sent to ApplicationMaster in heartbeat.  Here you ca= n do clean up activities for containers.

 

Hope this will help for you.  J!!

 

 

Thanks & Regards<= /p>

Rohith Sharma K S

 

From: Krishna Kishore Bonagi= ri [mailto:wri= te2kishore@gmail.com]
Sent: 07 April 2014 16:41
To: user= @hadoop.apache.org
Subject: Cleanup activity on YARN containers

 

Hi,

 

  Is there any callback kind of facility, in which I can writ= e some code to be executed on my container at the end of my application or = at the end of that particular container execution?

 

 I want to do some cleanup activities at the end of my applic= ation, and the clean up is not related to the localized resources that are = downloaded from HDFS.

 

Thanks,

Kishore

 

 

--_000_0EE80F6F7A98A64EBD18F2BE839C91156755D36Dszxeml512mbschi_--