hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gergo Repas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2185) Use pipes when localizing archives
Date Tue, 02 Jan 2018 12:10:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16307944#comment-16307944
] 

Gergo Repas commented on YARN-2185:
-----------------------------------

Thanks [~miklos.szegedi@cloudera.com] for the patch, I like this improvement. I have a couple
of comments and questions:
In {{FileUtil.runCommandOnStream()}}:
# the closing of {{process.getOutputStream()}} will not happen if there is an exception in
the first {{org.apache.commons.io.IOUtils.copy(inputStream, process.getOutputStream());}}
call.
# The process's outputstream may be closed before IOUtils.toString() has a chance to read
from it on the executor thread.
# The std error stream is not closed.
# {{org.apache.commons.io.IOUtils.copy(inputStream, process.getOutputStream());}} appears
twice: once before the process.waitFor() call, and once after - what's the reason for the
second call?

In {{RunJar.unJarAndSave()}} there is no need to use multiple try blocks, a single try-with
can handle multiple Closeable-s.

> Use pipes when localizing archives
> ----------------------------------
>
>                 Key: YARN-2185
>                 URL: https://issues.apache.org/jira/browse/YARN-2185
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.4.0
>            Reporter: Jason Lowe
>            Assignee: Miklos Szegedi
>         Attachments: YARN-2185.000.patch, YARN-2185.001.patch
>
>
> Currently the nodemanager downloads an archive to a local file, unpacks it, and then
removes it.  It would be more efficient to stream the data as it's being unpacked to avoid
both the extra disk space requirements and the additional disk activity from storing the archive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message