hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Szegedi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2185) Use pipes when localizing archives
Date Mon, 22 Jan 2018 23:40:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335173#comment-16335173
] 

Miklos Szegedi commented on YARN-2185:
--------------------------------------

Thank you for the review [~jlowe]!
bq. If the subprocess ends up producing too much output on either channel then this will deadlock. 
I verified and this scenario would use tar and gzip only and I made they do not have any output
in the normal scenario, so that should not block the pipe. However, indeed it is more future
proof to read both streams, so I fixed this.
I updated the patch reflecting all your comments.

 

> Use pipes when localizing archives
> ----------------------------------
>
>                 Key: YARN-2185
>                 URL: https://issues.apache.org/jira/browse/YARN-2185
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.4.0
>            Reporter: Jason Lowe
>            Assignee: Miklos Szegedi
>            Priority: Major
>         Attachments: YARN-2185.000.patch, YARN-2185.001.patch, YARN-2185.002.patch, YARN-2185.003.patch,
YARN-2185.004.patch, YARN-2185.005.patch, YARN-2185.006.patch, YARN-2185.007.patch, YARN-2185.008.patch
>
>
> Currently the nodemanager downloads an archive to a local file, unpacks it, and then
removes it.  It would be more efficient to stream the data as it's being unpacked to avoid
both the extra disk space requirements and the additional disk activity from storing the archive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message