hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Lilley <john.lil...@redpoint.net>
Subject RE: YARN tasks and child processes
Date Tue, 02 Jul 2013 15:41:31 GMT
Thanks, this is also good information.  But I was really asking if a child *process* that
was spawned by a task can persist, in addition to the data.

From: Devaraj k [mailto:devaraj.k@huawei.com]
Sent: Monday, July 01, 2013 11:50 PM
To: user@hadoop.apache.org
Subject: RE: YARN tasks and child processes

It is possible to persist the data by YARN task, you can choose whichever place you want to
If you choose to persist in HDFS, you need to take care deleting the data after using it.
 If you choose to write in local dir, you may write the data into the nm local dirs (i.e 'yarn.nodemanager.local-dirs'
configuration) accordingly with the app id & container id, and this will be cleaned up
after the app completion.  You need to make use of this persisted data before completing the

Devaraj k

From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: 02 July 2013 04:44
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: YARN tasks and child processes

Is it possible for a child process of a YARN task to persist after the task is complete? 
I am looking at an alternative to a YARN auxiliary process that may be simpler to implement,
if I can have a task spawn a process that persists for some time after the task finishes.

View raw message