hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3359) Yarn clients / AM should be able to provide config options to the RM / NM
Date Fri, 04 Nov 2011 20:55:52 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144330#comment-13144330

Daryn Sharp commented on MAPREDUCE-3359:

Although I helped contrib to this jira, I have concerns regarding its safety and hope it's
a temporary fix.

I feel the config is of questionable value since a misbehaving client may "forget" to cancel
its tokens.  The NN is holding tokens in memory so it could lead to a potential, and perhaps
unintentional, denial of service attack.

When tokens are shared between jobs, it's ambiguous as to when the tokens can be safely cancelled.
 How does a client know that other running or queued jobs are using the tokens?  If the client
intends to launch multiple jobs, but the client errors out, the tokens can't be cancelled
or "very bad" things will happen to the jobs already submitted.  Tasks will pound on the NN
every second with the bad token, and yarn tasks appear to run "forever" if rpc connections
fail.  In a test env, orphaned tasks had pounded on the NN every second for a *month*.

Allowing the RM to cancel tokens when the job completes, which implies tokens are good for
one and only one job submission, removes the ambiguity of when it's safe to cancel the tokens.
 This reduces the chance of a dos attack on the NN, and from a security perspective closes
the window of exposure vs. allowing tokens to linger until their lifetime expires.
> Yarn clients / AM should be able to provide config options to the RM / NM
> -------------------------------------------------------------------------
>                 Key: MAPREDUCE-3359
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3359
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Siddharth Seth
>            Assignee: Robert Joseph Evans
>         Attachments: MR-3359.txt
> The RM and NM do not read a job's configuration. Clients / AMs should however be able
to configure certain parameters for the RM/NM on a per app basis - like the Log Retention
policy, token cancellation.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message