hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2735) Setting default tmp directory for java createTempFile (java.io.tmpdir)
Date Mon, 10 Mar 2008 04:03:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576855#action_12576855

Amareshwari Sriramadasu commented on HADOOP-2735:

bq. Any consideration for Allen's comment?
aw> Although, it might be worthwhile setting the shell TMPDIR
aw> environment variable so that other languages have something
aw> they could use as well.

This is done as part of the patch. The environment variable,TMPDIR is set to the temp directory
created for boths pipes and streaming.

> Setting default tmp directory for java createTempFile (java.io.tmpdir)
> ----------------------------------------------------------------------
>                 Key: HADOOP-2735
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2735
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Koji Noguchi
>            Assignee: Amareshwari Sriramadasu
>            Priority: Critical
>             Fix For: 0.16.1
>         Attachments: patch-2735.txt, patch-2735.txt, patch-2735.txt, patch-2735.txt,
patch-2735.txt, patch-2735.txt, patch-2735.txt
> On our cluster, we've seen Pig(http://incubator.apache.org/pig/) filling up the /tmp
and failing. 
> (also inefficient since all the local tasks were spilling to the  same disk)
> Pig is simply using java api createTempFile, 
> http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html#createTempFile(java.lang.String,%20java.lang.String,%20java.io.File
> Can we add -Djava.io.tmpdir="./tmp" somewhere ?
> so that, 
> 1) Tasks can utilize all disks when using tmp
> 2) Any undeleted tmp files will be deleted by the tasktracker when task(job?) is done.
> The easiest way is to set it inside mapred.child.java.opts in the config, but this can
be overwritten if the users set their own task heapsize.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message