hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4700) ATS storage has one extra record each time the RM got restarted
Date Sat, 27 Feb 2016 19:41:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170704#comment-15170704

Vrushali C commented on YARN-4700:

Hi [~sjlee0] 
Let me add some more explanation.

bq. Wait, I think we're using the day timestamp for a reason as this table is supposed to
be a flow (daily) activity table.

Yes, the flow activity table indicates which apps were running at what time. If an event arrives
late (or in this case, a replay causes it arrive at a later time), it still belongs to the
day the app ran on. So the entry for that flow should go into THAT older day's row, hence
we should use the event timestamp.

bq.  And some considerations are given to long running apps that will cross the day boundaries.

For long running apps, we would most likely be making a snapshot entry that belongs to the
day on which the app was running.

bq. I'd like us to stick with that unless there is a compelling reason not to?
So we are not changing the semantics here by using the event timestamp. We are actually making
an explicit entry for the actual day on which the app ran, rather than relying on when the
event reached the backend.

We can chat further on monday. 

> ATS storage has one extra record each time the RM got restarted
> ---------------------------------------------------------------
>                 Key: YARN-4700
>                 URL: https://issues.apache.org/jira/browse/YARN-4700
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Li Lu
>            Assignee: Naganarasimha G R
>              Labels: yarn-2928-1st-milestone
> When testing the new web UI for ATS v2, I noticed that we're creating one extra record
for each finished application (but still hold in the RM state store) each time the RM got
restarted. It's quite possible that we add the cluster start timestamp into the default cluster
id, thus each time we're creating a new record for one application (cluster id is a part of
the row key). We need to fix this behavior, probably by having a better default cluster id.

This message was sent by Atlassian JIRA

View raw message