hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable
Date Fri, 07 Sep 2012 06:30:07 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450412#comment-13450412
] 

Karthik Kambatla commented on MAPREDUCE-4643:
---------------------------------------------

The two aspects of making the job-history cleanup period configurable seem to be: 
# unit of time to configure it on (minutes vs days)
# directory naming to reflect the creation time

For the directory naming, we should be able to leave it as yyyy/mm/dd format. If the cleanup
period is in hours/minutes, we can read the file modification time and use that to do the
comparison. Trunk currently uses the modification time even for higher values of cleanup period
(days) and has a TODO note that we should use the directory structure instead to reduce the
load on HDFS.

With regards to the unit of time for configuration, minutes seems to be the better option.

Thoughts?
                
> Make job-history cleanup-period configurable
> --------------------------------------------
>
>                 Key: MAPREDUCE-4643
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>    Affects Versions: 1.0.3, 2.0.0-alpha
>            Reporter: Karthik Kambatla
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default.
The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be
configurable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message