hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6889) Provide an API Job#close to shutdown MR client services.
Date Tue, 16 May 2017 15:26:04 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rohith Sharma K S updated MAPREDUCE-6889:
-----------------------------------------
    Description: 
ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer initialization.
If writer is not closed, then there is possibility of OOM see YARN-5438 fixes closing FS object.


TimelineClient is used by YarnClient. So all the user who uses YarnClient with ATS1.5 need
to stop service properly. Otherwise there is big chance of FS object leak. 

Of course MR uses YARN client submit job. If MR do not stop YarnClient then there is FS object
leak. 

JobClient provides a API to stop all these service using *JobClient#close*. But many MR clients
uses *Job* object to submit a job. But  do not stop started services by default. 

So, provide a API *Job#close* to shutdown MR client services. This API can be utilized by
caller explicitly to stop service which avoids FS leak. 

  was:
ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer initialization.
If writer is not closed, then there is possibility of OOM see YARN-5438 fixes closing FS object.


TimelineClient is used by YarnClient. So all the user who uses YarnClient with ATS1.5 need
to stop service properly. Otherwise there is big chance of FS object leak. 

Of course MR uses YARN client submit job. If MR do not stop YarnClient then there is FS object
leak. 

JobClient provides a API to stop all these service using *JobClient#close*. But many MR clients
uses *Job* object to submit a job. But  do not stop started services by default. So, Job class
should provide a API to close the services very similar to JobClient#close. 


> Provide an API Job#close to shutdown MR client services.
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-6889
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6889
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>         Attachments: MAPREDUCE-6889.0001.patch
>
>
> ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer initialization.
If writer is not closed, then there is possibility of OOM see YARN-5438 fixes closing FS object.

> TimelineClient is used by YarnClient. So all the user who uses YarnClient with ATS1.5
need to stop service properly. Otherwise there is big chance of FS object leak. 
> Of course MR uses YARN client submit job. If MR do not stop YarnClient then there is
FS object leak. 
> JobClient provides a API to stop all these service using *JobClient#close*. But many
MR clients uses *Job* object to submit a job. But  do not stop started services by default.

> So, provide a API *Job#close* to shutdown MR client services. This API can be utilized
by caller explicitly to stop service which avoids FS leak. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message