hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Miklos Szegedi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2185) Use pipes when localizing archives
Date Wed, 24 Jan 2018 21:47:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338279#comment-16338279
] 

Miklos Szegedi commented on YARN-2185:
--------------------------------------

Thank you, [~jlowe]. I updated the patch.
{quote}makeShellPath is a pre-existing, public function,...
{quote}
If we want to be conservative with this jira I would be more conservative with security rather
than compatibility. I used both Windows and Linux but I have never seen a legitimate path
with ' or ". That said, I would rather throw an exception in these cases rather than trying
to be backward compatible for existing malware. :) I changed the patch to have a new makeSecureShellPath
function, and I ignore Windows, since it is dead code right now, and it is not so trivial
to avoid code injection.
{quote}Attempting to get the futures from the executor could result in an ExecutionException...
{quote}
I caught the only IOException thrown there but anyone can change the code later, so I refactored
the it a little bit to protect this scenario. I also log now on the fly, to avoid OOM errors.
Let me know, what you think.

> Use pipes when localizing archives
> ----------------------------------
>
>                 Key: YARN-2185
>                 URL: https://issues.apache.org/jira/browse/YARN-2185
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.4.0
>            Reporter: Jason Lowe
>            Assignee: Miklos Szegedi
>            Priority: Major
>         Attachments: YARN-2185.000.patch, YARN-2185.001.patch, YARN-2185.002.patch, YARN-2185.003.patch,
YARN-2185.004.patch, YARN-2185.005.patch, YARN-2185.006.patch, YARN-2185.007.patch, YARN-2185.008.patch,
YARN-2185.009.patch, YARN-2185.010.patch
>
>
> Currently the nodemanager downloads an archive to a local file, unpacks it, and then
removes it.  It would be more efficient to stream the data as it's being unpacked to avoid
both the extra disk space requirements and the additional disk activity from storing the archive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message