hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3251) JobClient should have an option to only to talk to RM+HistoryServer to get job status
Date Mon, 24 Oct 2011 18:32:33 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134331#comment-13134331

Milind Bhandarkar commented on MAPREDUCE-3251:


Which method in RMAppAttempt will be used for reporting JobStatus ?

I believe, one will have to get the AM container from the RM, and then inquire status on that
Container, right ?

The ContainerStatus only contains ContainerId and State. There is not job-specific information
there. This is where I was proposing a K-V query style interface. (The reason it needs to
be K-V style, is if/when the MR AM (i.e. JT) evolves into running multiple MR jobs (or a job
chain as in JobControl), it will come it handy to only get the status of a specific job instead
of an entire chain.)

> JobClient should have an option to only to talk to RM+HistoryServer to get job status
> -------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-3251
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Anupam Seth
>            Priority: Blocker
>             Fix For: 0.23.0
> In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN,
AM can be launched on any port and the client will have to have ACL open to that port to talk
to AM and get the job status. When the client is within the same grid network access to AM
is not a problem. But some applications may have one installation per set of clusters and
may launch jobs even across such sets (on job trackers in another set of clusters). For that
to work only the JT port needs to be open currently. In case of YARN, all ports will have
to be opened up for things to work. That would be a security no-no.
> There are two possible solutions:
>   1) Make the job client only talk to RM (as an option) to get the job status. 
>   2) Limit the range of ports AM can listen on.
> Option 2) may not be favorable as there is no direct OS API to find a free port.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message