hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joep Rottinghuis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5742) Serve aggregated logs of historical apps from timeline service
Date Thu, 20 Oct 2016 18:45:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592635#comment-15592635

Joep Rottinghuis commented on YARN-5742:

We should really carefully consider whether serving yarn application logs is a timeline service
I'd argue that belongs in a separate Yarn service. It would be perfectly acceptable to have
that separate service store metadata (about the current log location of tasks, whether that
is on local disk, or aggregated to a location such as HDFS) in the timeline service, but I
think the serving itself doesn't below here.

Providing an api that can read files from HDFS and stream them out would be opening security
concerns, would duplicate efforts from services such as WebHDFS/HttpFS and all their concerns.
The HttpFS approach of a central pool of nodes serving data from HDFS has been superseded
by a distributed WebHDFS approach. Note by the way that WebHDFS as it stands today still has
some compatibility challenges with HDFS federation. Both of these general approaches of serving
HDFS data have to deal with proxying user requests and correctly limiting visibility of HDFS
files to the users with the appropriate access.

> Serve aggregated logs of historical apps from timeline service
> --------------------------------------------------------------
>                 Key: YARN-5742
>                 URL: https://issues.apache.org/jira/browse/YARN-5742
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Rohith Sharma K S
>         Attachments: YARN-5742-POC-v0.patch

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message