hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: temporary folders for YARN tasks
Date Tue, 02 Jul 2013 06:12:44 GMT
LocalDirAllocator should help with this.  You can look through MapReduce
code to see how it's used.

-Sandy


On Mon, Jul 1, 2013 at 11:01 PM, Devaraj k <devaraj.k@huawei.com> wrote:

>  You can make use of this configuration to do the same. ****
>
> ** **
>
> <property>****
>
>     <description>List of directories to store *localized* files in. An ***
> *
>
>       application's *localized* file directory will be found in:****
>
>       ${yarn.nodemanager.local-*dirs*}/*usercache*/${user}/*appcache*
> /application_${*appid*}.****
>
>       Individual containers' work directories, called container_${*contid*},
> will****
>
>       be *subdirectories* of this.****
>
>    </description>****
>
>     <name>yarn.nodemanager.local-*dirs*</name>****
>
>     <value>${hadoop.tmp.dir}/*nm*-local-*dir*</value>****
>
>   </property>****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net]
> *Sent:* 02 July 2013 02:08
> *To:* user@hadoop.apache.org
> *Subject:* temporary folders for YARN tasks****
>
> ** **
>
> When a YARN app and its tasks wants to write temporary files, how does it
> know where to write the files?  ****
>
> I am assuming that each task has some temporary space available, and I
> hope it is available across multiple disk volumes for parallel performance.
> ****
>
> Are those files cleaned up automatically after task exit?****
>
> If I want to give lifetime control of the files to an auxiliary service
> (along the lines of MR shuffle passing files to the aux service), how would
> I do that, and would that entail different file locations?****
>
> Thanks****
>
> John****
>
> ** **
>
> ** **
>

Mime
View raw message