pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2591) Unit tests should not write to /tmp but respect java.io.tmpdir
Date Mon, 11 Mar 2013 17:35:13 GMT

    [ https://issues.apache.org/jira/browse/PIG-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599030#comment-13599030
] 

Cheolsoo Park commented on PIG-2591:
------------------------------------

[~jarcec] and [~prkommireddi], I thought about this more last night. I withdraw my agreement
on Jarcec's patch.
* I don't think we want to have the following method:
{code}
public static String getTemporaryDirectory() {
    return getTemporaryDirectory(System.getProperties(), random);
}
{code}
Let's say that we set "pig.temp.dir" to "/x" in system properties (e.g. <sysproperty>
in build.xml). But FilLocalizer.getTemporaryPath() is not effected by this because "pig.temp.dir"
is not set in PigContext. So Pig will still generate intermediate data under "file:///tmp".
Now if we do "store foo into TmpUtil.getTemporaryDirectory()", unit tests will generate temporary
files into two directories: output files into "/x" and intermediate files into "/tmp".
{quote}
The direct usage of FileLocalizer is not simple as it requires a lot of initialization (for
example that class needs valid PigContext).
{quote}
In fact, this is good because that's how we guarantee all the temporary files go into a single
directory controlled by PigContext. So we should use FilLocalizer.getTemporaryPath().
* I was wrong about setting "pig.temp.dir" in system properties in build.xml for unit tests.
In fact, it won't be propagated to PigContext. So we should set "pig.temp.dir" in PigContext.
* All of these are only applicable to local mode (i.e. file://). If unit tests run on mini
cluster, we shouldn't set "pig.temp.dir". Currently, mini cluster already generates files
under "./build", so we don't need to worry about them.
* We should use File.createTempFile() for local temporary files that are not generated by
Pig. (e.g. Some tests generate input files on local and copy them to mini cluster.) Then,
"java.io.tmpdir" will be automatically honored.

I think I covered all the cases. Please let me know what you think.
                
> Unit tests should not write to /tmp but respect java.io.tmpdir
> --------------------------------------------------------------
>
>                 Key: PIG-2591
>                 URL: https://issues.apache.org/jira/browse/PIG-2591
>             Project: Pig
>          Issue Type: Bug
>          Components: tools
>            Reporter: Thomas Weise
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 0.12
>
>         Attachments: bugPIG-2591.patch, PIG-2495.patch
>
>
> Several tests use /tmp but should derive temporary file location from java.io.tmpdir
to avoid side effects (java.io.tmpdir is already set to a test run specific location in build.xml)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message