Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Thu, 26 Sep 2013 01:04:02 +0000 (UTC)
From: "shanyu zhao (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12669371.1379548257838.21437.1380157442842@arcas>
In-Reply-To: <JIRA.12669371.1379548257838@arcas>
References: <JIRA.12669371.1379548257838@arcas>
Subject: [jira] [Commented] (YARN-1219) FSDownload changes file suffix
 making FileUtil.unTar() throw exception
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778330#comment-13778330 ] 

shanyu zhao commented on YARN-1219:
-----------------------------------

The impact of this bug is that Yarn will fail to download any .tar.gz or .tgz files as resources on Windows platform.

[~ojoshi] Thank you for reviewing this patch. I think the ".tmp" file name suffix is only used to make sure the temporary archive file use a different name than the archive file name itself. So the file name of dCopy has to be different from sCopy. Because the final unarchived files need to be put in a folder whose name is the exactly the original archive file name. 

Here is what happens when a container downloads an archived resource file, let's say the file name is "stage-3.tar.gz" and the destination dir is "filecache/10".
  a. Create destDirPath folder "filecache/10"
  b. Create dst_work folder "filecache/10_tmp"
  c. Copy archive file to dTmp "filecache/10_tmp/stage-3.tar.gz.tmp"
  d. Unpack dTmp "filecache/10_tmp/stage-3.tar.gz.tmp" into dFinal "filecache/10_tmp/stage-3.tar.gz/"
  e. Rename folder dst_work "filecache/10_tmp" to destDirPath "filecache/10"
So after all these steps, the unzipped unarchived files can be found in "filecache/10/stage-3.tar.gz/*"

I also checked in Hadoop 1.x the file name being downloaded from HDFS does not have the ".tmp" suffix. It is using the original file name as the file name to be downloaded.

I didn't see anywhere in code to treat the ".tmp" file differently. If you know please let me know. If the original author only used a suffix to make sure the name is different than the original file name, it doesn't seem to be worth it to add an unnecessary and error-prone rename operations just to keep the temporary file name suffix.
                
> FSDownload changes file suffix making FileUtil.unTar() throw exception
> ----------------------------------------------------------------------
>
>                 Key: YARN-1219
>                 URL: https://issues.apache.org/jira/browse/YARN-1219
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 3.0.0, 2.1.1-beta, 2.1.2-beta
>            Reporter: shanyu zhao
>            Assignee: shanyu zhao
>             Fix For: 2.1.2-beta
>
>         Attachments: YARN-1219.patch
>
>
> While running a Hive join operation on Yarn, I saw exception as described below. This is caused by FSDownload copy the files into a temp file and change the suffix into ".tmp" before unpacking it. In unpack(), it uses FileUtil.unTar() which will determine if the file is "gzipped" by looking at the file suffix:
> {code}
> boolean gzipped = inFile.toString().endsWith("gz");
> {code}
> To fix this problem, we can remove the ".tmp" in the temp file name.
> Here is the detailed exception:
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
> 	at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
> 	at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
> 	at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
> 	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
> 	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira