hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6889) FileSystem leak when Job#submit() used when ATS1.5 enabled
Date Tue, 16 May 2017 15:12:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012554#comment-16012554

Rohith Sharma K S commented on MAPREDUCE-6889:

bq. It would be good to have a unit test that verifies closing the job closes the cluster
(or at least the client protocol provider). Also should cluster be set to null after we call
close on it?
make sense, I will add a test case.

bq. Callers would have to be updated to invoke the new close method at the appropriate time,
so the headline is a bit misleading if we marked this as fixed.
thats true. Caller must close the Job object to avoid leak. I will also update JIRA summary.

> FileSystem leak when Job#submit() used when ATS1.5 enabled
> ----------------------------------------------------------
>                 Key: MAPREDUCE-6889
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6889
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>         Attachments: MAPREDUCE-6889.0001.patch
> ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer initialization.
If writer is not closed, then there is possibility of OOM see YARN-5438 fixes closing FS object.

> TimelineClient is used by YarnClient. So all the user who uses YarnClient with ATS1.5
need to stop service properly. Otherwise there is big chance of FS object leak. 
> Of course MR uses YARN client submit job. If MR do not stop YarnClient then there is
FS object leak. 
> JobClient provides a API to stop all these service using *JobClient#close*. But many
MR clients uses *Job* object to submit a job. But  do not stop started services by default.
So, Job class should provide a API to close the services very similar to JobClient#close.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message