hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2185) Use pipes when localizing archives
Date Thu, 18 Jan 2018 22:19:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331335#comment-16331335
] 

Robert Kanter commented on YARN-2185:
-------------------------------------

Here's some comments:
# The do-while loop in {{RunJar#unJar}} can be made into a for-loop, which is probably more
clear and avoids the infinite loop and break.  {{for(JarEntry entry = jar.getNextJarEntry();
entry == null; entry = jar.getNextJarEntry())}}
#- Similarly with {{FileUtil#unZip}}
# In {{FSDownload}}, it has {{downloadAndUnpack}} catch {{Exception}} to wrap and throw it
in a {{RuntimeException}}, which is then caught and thrown as a {{YarnException}} in {{verifyAndCopy}}.
 Why not simply have {{downloadAndUnpack}} throw a {{YarnException}} directly instead of a
{{RuntimeException}}, cutting out one level of wrapping?  {{downloadAndUnpack}} is a new and
{{private}} method, so there's no worry about needing to make it not declare a throwable.
# In the untar command, shouldn't we use {{&&}} instead of {{;}}?  With {{;}}, it'll
continue even if the first command fails.  It looks like you already agreed with [~jlowe]
to make this change?  We should also fix the existing {{unTarUsingTar}} method too.
# There's a number of utility functions where you added a second version that operates on
streams instead of files.  We should rename the relevant arguments for clarity, like you did
in {{unTarUsingJava}} ({{inFile}} --> {{inputStream}}).

> Use pipes when localizing archives
> ----------------------------------
>
>                 Key: YARN-2185
>                 URL: https://issues.apache.org/jira/browse/YARN-2185
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.4.0
>            Reporter: Jason Lowe
>            Assignee: Miklos Szegedi
>            Priority: Major
>         Attachments: YARN-2185.000.patch, YARN-2185.001.patch, YARN-2185.002.patch, YARN-2185.003.patch,
YARN-2185.004.patch, YARN-2185.005.patch, YARN-2185.006.patch
>
>
> Currently the nodemanager downloads an archive to a local file, unpacks it, and then
removes it.  It would be more efficient to stream the data as it's being unpacked to avoid
both the extra disk space requirements and the additional disk activity from storing the archive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message