pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Coveney (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-2796) Local temporary paths are not always valid HDFS path names.
Date Thu, 18 Oct 2012 17:46:04 GMT

    [ https://issues.apache.org/jira/browse/PIG-2796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479176#comment-13479176
] 

Jonathan Coveney commented on PIG-2796:
---------------------------------------

I feel like the real fix is to use FilerLocalizer.getTemporaryPath(PigContext), no? This gives
you a temporary path in HDFS. We can make sure that works on Windows (and should)
                
> Local temporary paths are not always valid HDFS path names.
> -----------------------------------------------------------
>
>                 Key: PIG-2796
>                 URL: https://issues.apache.org/jira/browse/PIG-2796
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.10.0
>            Reporter: John Gordon
>            Assignee: John Gordon
>             Fix For: 0.11
>
>         Attachments: 0006-Local-Remote-file-mapping-for-tests-with-temps.patch
>
>
> A number of pig scripts follow the pattern:
> File tempFile = File.createTempFile("this", ".txt");
> copyFromLocalToCluster (tempFile.to_string(), tempFile.to_string());
> tempFile.delete();
> The goal, here, seems to be to generate a temp filename to avoid issues on the next run
if the file doesn't get cleaned up.  The problem is that File.createTempFile on Windows creates
files with names like C:\users\myuser\App data\local\temp\file.txt.  The problem is that ":"
is not a valid DFS character and so the put fails.
> The easy fix on this is to remove colons from the path before upload.  Then we get something
like C\users\myuser\App data\local\temp\file.txt which is a valid DFS pathname with minimal
impact to the tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message