hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joep Rottinghuis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5742) Serve aggregated logs of historical apps from timeline service
Date Thu, 20 Oct 2016 18:45:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592635#comment-15592635
] 

Joep Rottinghuis commented on YARN-5742:
----------------------------------------

We should really carefully consider whether serving yarn application logs is a timeline service
concern.
I'd argue that belongs in a separate Yarn service. It would be perfectly acceptable to have
that separate service store metadata (about the current log location of tasks, whether that
is on local disk, or aggregated to a location such as HDFS) in the timeline service, but I
think the serving itself doesn't below here.

Providing an api that can read files from HDFS and stream them out would be opening security
concerns, would duplicate efforts from services such as WebHDFS/HttpFS and all their concerns.
The HttpFS approach of a central pool of nodes serving data from HDFS has been superseded
by a distributed WebHDFS approach. Note by the way that WebHDFS as it stands today still has
some compatibility challenges with HDFS federation. Both of these general approaches of serving
HDFS data have to deal with proxying user requests and correctly limiting visibility of HDFS
files to the users with the appropriate access.

> Serve aggregated logs of historical apps from timeline service
> --------------------------------------------------------------
>
>                 Key: YARN-5742
>                 URL: https://issues.apache.org/jira/browse/YARN-5742
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Varun Saxena
>            Assignee: Rohith Sharma K S
>         Attachments: YARN-5742-POC-v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message