hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4700) ATS storage has one extra record each time the RM got restarted
Date Sat, 27 Feb 2016 05:18:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170376#comment-15170376
] 

Varun Saxena commented on YARN-4700:
------------------------------------

bq.  I think we're using the day timestamp for a reason as this table is supposed to be a
flow (daily) activity table. And some considerations are given to long running apps that will
cross the day boundaries.
Let us assume RM does not restart. In that case, we will only get the start event and finish
event once each. In that case, event timestamp will be close to current timestamp.
And if those are the only events we get, issue with long running apps(extending over more
than 2 days) will anyways be there. For instance if  we get start event on day 1 and finish
event on day 3 and if there is no other app for this flow, this will lead to no activity on
day 2 even if we use current timestamp..
There was YARN-4069 which was filed for this issue and its with me. I was thinking of scheduling
a global timer in RM which can emit  ATS events for all the running apps at a certain point
of time.This should resolve long running app issue. This is not marked for 1st milestone though
so no progress made.

> ATS storage has one extra record each time the RM got restarted
> ---------------------------------------------------------------
>
>                 Key: YARN-4700
>                 URL: https://issues.apache.org/jira/browse/YARN-4700
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Li Lu
>            Assignee: Naganarasimha G R
>              Labels: yarn-2928-1st-milestone
>
> When testing the new web UI for ATS v2, I noticed that we're creating one extra record
for each finished application (but still hold in the RM state store) each time the RM got
restarted. It's quite possible that we add the cluster start timestamp into the default cluster
id, thus each time we're creating a new record for one application (cluster id is a part of
the row key). We need to fix this behavior, probably by having a better default cluster id.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message