hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anupam Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3251) Network ACLs can prevent some clients to talk to MR ApplicationMaster
Date Fri, 09 Dec 2011 17:23:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166329#comment-13166329

Anupam Seth commented on MAPREDUCE-3251:

bq. YOu are returning a jobhistory proxy when the switch is turned on? Why is that?

Thanks Mahadev! I am returning the jobhistory proxy because the attempt to connect to the
AM happens inside the call to getProxy, which is expecting some proxy back. In case, the network
ACL is disabled, this behavior of returning the JH proxy is identical to the case when the
connection to the AM fails for some unknown reason. The JH proxy is returned by get Proxy()
with the application state set as RUNNING so that the user can still get a meaningful response
in this condition. 

      } catch (IOException e) {
        //possibly the AM has crashed
        //there may be some time before AM is restarted
        //keep retrying by getting the address from RM
        LOG.info("Could not connect to " + serviceAddr +
        ". Waiting for getting the latest AM address...");
        application = rm.getApplicationReport(appId);
        if (application == null) {
          LOG.debug("Could not get Job info from RM for job " + jobId
              + ". Redirecting to job history server.");
          return checkAndGetHSProxy(null, JobState.RUNNING);

If that doesn't make sense, what would be a good alternative to return?

> Network ACLs can prevent some clients to talk to MR ApplicationMaster
> ---------------------------------------------------------------------
>                 Key: MAPREDUCE-3251
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Anupam Seth
>            Assignee: Anupam Seth
>            Priority: Critical
>             Fix For: 0.23.1
>         Attachments: MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251-branch_0_23.patch,
> In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN,
AM can be launched on any port and the client will have to have ACL open to that port to talk
to AM and get the job status. When the client is within the same grid network access to AM
is not a problem. But some applications may have one installation per set of clusters and
may launch jobs even across such sets (on job trackers in another set of clusters). For that
to work only the JT port needs to be open currently. In case of YARN, all ports will have
to be opened up for things to work. That would be a security no-no.
> There are two possible solutions:
>   1) Make the job client only talk to RM (as an option) to get the job status. 
>   2) Limit the range of ports AM can listen on.
> Option 2) may not be favorable as there is no direct OS API to find a free port.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message