hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7265) Hadoop Server Log Correlation
Date Thu, 28 Sep 2017 19:51:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184738#comment-16184738

Jason Lowe commented on YARN-7265:

Is this really appropriate to be built directly into Hadoop?  A sophisticated log analytics
and log correlation system seems like something that would be better built on top of Hadoop
as a separate project.  Hive, Pig, Oozie, HBase, Ambari, etc. are all important parts of the
Hadoop stack, but there are good reasons why they are not built into Hadoop.  This seems like
another thing that would be very useful as part of the Hadoop stack but not necessary to build
in directly.

> Hadoop Server Log Correlation  
> -------------------------------
>                 Key: YARN-7265
>                 URL: https://issues.apache.org/jira/browse/YARN-7265
>             Project: Hadoop YARN
>          Issue Type: Wish
>          Components: log-aggregation
>            Reporter: Tanping Wang
> Hadoop has many server logs, yarn tasks logs, node manager logs, HDFS logs..   There
are also a lot of different ways can be used to expose the logs, build relationship horizontally
to correlate the logs or search the logs by keyword. There is a need for a default and yet
convenient  logging analytics mechanism in Hadoop itself that at least covers all the server
logs of Hadoop.  This log analytics system can correlate the Hadoop server logs by grouping
them by various dimensions including application ID,  task ID, job ID or node ID etc.   The
raw logs with correlation can be easily accessed by the application developer or cluster administrator
via web page for managing and debugging.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message