hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
Date Thu, 23 Jul 2015 00:46:05 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637998#comment-14637998
] 

Allen Wittenauer commented on MAPREDUCE-6415:
---------------------------------------------

bq.  The shell inherits the env of the NodeManager as a base. HADOOP_HOME should be defined
for the NM, so it ends up in env of the shell.

a) This is only true for Windows.  Unix has been using HADOOP_PREFIX since 0.21.  If it's
being defined, it's not by the bash code that starts the NM that ships with Apache Hadoop.

b) I'm unsure if LCE actually inherits all of the shell environment or only specific variables.

> Create a tool to combine aggregated logs into HAR files
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-6415
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>    Affects Versions: 2.8.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415_branch-2_prelim_001.patch,
MAPREDUCE-6415_prelim_001.patch
>
>
> While we wait for YARN-2942 to become viable, it would still be great to improve the
aggregated logs problem.  We can write a tool that combines aggregated log files into a single
HAR file per application, which should solve the too many files and too many blocks problems.
 See the design document for details.
> See YARN-2942 for more context.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message