hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhang Wei (JIRA)" <j...@apache.org>
Subject [jira] [Work started] (MAPREDUCE-6283) MRHistoryServer log files management optimization
Date Fri, 20 Mar 2015 03:39:38 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Work on MAPREDUCE-6283 started by Zhang Wei.
--------------------------------------------
> MRHistoryServer log files management optimization
> -------------------------------------------------
>
>                 Key: MAPREDUCE-6283
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6283
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobhistoryserver
>            Reporter: Zhang Wei
>            Assignee: Zhang Wei
>            Priority: Minor
>   Original Estimate: 2,016h
>  Remaining Estimate: 2,016h
>
> In some heavy computation clusters, there might be a potential hdfs small files problem.
The continually submitted MR jobs will create millions of log files (include the application
master logs and application logs which are aggregated to hdfs by nodemanager). This optimization
design helps to reduce the numbers of log files by merging them into bigger ones.
> Get the details from the design doc which I will upload later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message