hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anupam Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3251) Network ACLs can prevent some clients to talk to MR ApplicationMaster
Date Thu, 15 Dec 2011 20:55:30 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170486#comment-13170486
] 

Anupam Seth commented on MAPREDUCE-3251:
----------------------------------------

bq. Anupam, did you do a real cluster-test or an integration test?
@Vinod, yes I did. Here is console output upon disabling the ACL and running a word count
job. I think I see the intent of what you are saying, and it will probably definitely be cleaner,
but for some reason, it isn't as broken I think.

@Mahadev, I will upload a new patch with the suggestions you have outlined in pursuance of
the above comments.

11/12/15 20:41:32 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
11/12/15 20:41:32 WARN conf.Configuration: mapred.used.genericoptionsparser is deprecated.
Instead, use mapreduce.client.genericoptionsparser.used
11/12/15 20:41:32 INFO input.FileInputFormat: Total input paths to process : 1
11/12/15 20:41:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
11/12/15 20:41:32 WARN snappy.LoadSnappy: Snappy native library not loaded
11/12/15 20:41:32 INFO mapreduce.JobSubmitter: number of splits:1
11/12/15 20:41:33 INFO mapred.ResourceMgrDelegate: Submitted application application_1323981651676_0001
to ResourceManager at <hostname>/98.139.92.65:8040
11/12/15 20:41:33 INFO mapreduce.Job: Running job: job_1323981651676_0001
11/12/15 20:41:42 INFO mapred.ClientServiceDelegate: AppId: application_1323981651676_0001
# reserved containers: 0 # used containers: 1 Needed resources (memory): 2048 Reserved resources
(memory): 0 Used resources (memory): 2048 Diagnostics:  Start time: 1323981693246 Finish time:
0 Host: <hostname> Name: word count Orig. tracking url: <hostname>:50256 Queue:
default RPC port: 55191 Tracking url: <hostname>:8088/proxy/application_1323981651676_0001/
User: <user> Client token: null Final appl. status: UNDEFINED Yarn appl. state: RUNNING
....
....
....
11/12/15 20:41:56 INFO mapred.ClientServiceDelegate: Network ACL closed to AM for job job_1323981651676_0001.
Redirecting to job history server.
11/12/15 20:41:56 WARN mapred.ClientServiceDelegate: Job History Server is not configured
or job information not yet available on History Server.
11/12/15 20:41:56 INFO mapred.ClientServiceDelegate: AppId: application_1323981651676_0001
# reserved containers: 0 # used containers: 1 Needed resources (memory): 2048 Reserved resources
(memory): 0 Used resources (memory): 2048 Diagnostics:  Start time: 1323981693246 Finish time:
0 Host: <hostname> Name: word count Orig. tracking url: <hostname>:50256 Queue:
default RPC port: 55191 Tracking url: <hostname>:8088/proxy/application_1323981651676_0001/
User: <user> Client token: null Final appl. status: UNDEFINED Yarn appl. state: RUNNING
11/12/15 20:41:56 INFO mapred.ClientServiceDelegate: Network ACL closed to AM for job job_1323981651676_0001.
Redirecting to job history server.
11/12/15 20:41:56 WARN mapred.ClientServiceDelegate: Job History Server is not configured
or job information not yet available on History Server.
11/12/15 20:41:57 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED.
Redirecting to job history server
11/12/15 20:41:57 WARN mapred.ClientServiceDelegate: Job History Server is not configured
or job information not yet available on History Server.
11/12/15 20:41:57 INFO mapreduce.Job: Job job_1323981651676_0001 completed successfully
11/12/15 20:41:57 INFO mapreduce.Job: Counters: 0

                
> Network ACLs can prevent some clients to talk to MR ApplicationMaster
> ---------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3251
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Anupam Seth
>            Assignee: Anupam Seth
>            Priority: Critical
>             Fix For: 0.23.1
>
>         Attachments: MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251-branch_0_23.patch,
MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251-branch_0_23.patch, MAPREDUCE-3251_branch-0_23_preliminary.txt
>
>
> In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN,
AM can be launched on any port and the client will have to have ACL open to that port to talk
to AM and get the job status. When the client is within the same grid network access to AM
is not a problem. But some applications may have one installation per set of clusters and
may launch jobs even across such sets (on job trackers in another set of clusters). For that
to work only the JT port needs to be open currently. In case of YARN, all ports will have
to be opened up for things to work. That would be a security no-no.
> There are two possible solutions:
>   1) Make the job client only talk to RM (as an option) to get the job status. 
>   2) Limit the range of ports AM can listen on.
> Option 2) may not be favorable as there is no direct OS API to find a free port.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message