hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shanyu zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1219) FSDownload changes file suffix making FileUtil.unTar() throw exception
Date Thu, 26 Sep 2013 01:04:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13778330#comment-13778330

shanyu zhao commented on YARN-1219:

The impact of this bug is that Yarn will fail to download any .tar.gz or .tgz files as resources
on Windows platform.

[~ojoshi] Thank you for reviewing this patch. I think the ".tmp" file name suffix is only
used to make sure the temporary archive file use a different name than the archive file name
itself. So the file name of dCopy has to be different from sCopy. Because the final unarchived
files need to be put in a folder whose name is the exactly the original archive file name.

Here is what happens when a container downloads an archived resource file, let's say the file
name is "stage-3.tar.gz" and the destination dir is "filecache/10".
  a. Create destDirPath folder "filecache/10"
  b. Create dst_work folder "filecache/10_tmp"
  c. Copy archive file to dTmp "filecache/10_tmp/stage-3.tar.gz.tmp"
  d. Unpack dTmp "filecache/10_tmp/stage-3.tar.gz.tmp" into dFinal "filecache/10_tmp/stage-3.tar.gz/"
  e. Rename folder dst_work "filecache/10_tmp" to destDirPath "filecache/10"
So after all these steps, the unzipped unarchived files can be found in "filecache/10/stage-3.tar.gz/*"

I also checked in Hadoop 1.x the file name being downloaded from HDFS does not have the ".tmp"
suffix. It is using the original file name as the file name to be downloaded.

I didn't see anywhere in code to treat the ".tmp" file differently. If you know please let
me know. If the original author only used a suffix to make sure the name is different than
the original file name, it doesn't seem to be worth it to add an unnecessary and error-prone
rename operations just to keep the temporary file name suffix.
> FSDownload changes file suffix making FileUtil.unTar() throw exception
> ----------------------------------------------------------------------
>                 Key: YARN-1219
>                 URL: https://issues.apache.org/jira/browse/YARN-1219
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 3.0.0, 2.1.1-beta, 2.1.2-beta
>            Reporter: shanyu zhao
>            Assignee: shanyu zhao
>             Fix For: 2.1.2-beta
>         Attachments: YARN-1219.patch
> While running a Hive join operation on Yarn, I saw exception as described below. This
is caused by FSDownload copy the files into a temp file and change the suffix into ".tmp"
before unpacking it. In unpack(), it uses FileUtil.unTar() which will determine if the file
is "gzipped" by looking at the file suffix:
> {code}
> boolean gzipped = inFile.toString().endsWith("gz");
> {code}
> To fix this problem, we can remove the ".tmp" in the temp file name.
> Here is the detailed exception:
> org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
> 	at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
> 	at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
> 	at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
> 	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
> 	at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message