hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3942) Timeline store to read events from HDFS
Date Thu, 30 Jul 2015 18:19:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648057#comment-14648057
] 

Jason Lowe commented on YARN-3942:
----------------------------------

The logs are created based on app attempt.  It helps avoid the split-brain, double-writer
issue where the previous attempt is still running when the RM expires it (e.g.: due to network
cut) and decides to launch another.  The files are stored and looked up in a directory that
is named after the application ID, and the entity files within that directory are stored based
on application attempt ID.  I don't think the latter is crucial to use the app attempt ID
and the reader is not relying on the attempt ID from those files, but it was a simple way
to avoid colliding with previous attempts and having the reader process the files in attempt
order.

> Timeline store to read events from HDFS
> ---------------------------------------
>
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942.001.patch
>
>
> This adds a new timeline store plugin that is intended as a stop-gap measure to mitigate
some of the issues we've seen with ATS v1 while waiting for ATS v2.  The intent of this plugin
is to provide a workable solution for running the Tez UI against the timeline server on a
large-scale clusters running many thousands of jobs per day.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message