hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4696) EntityGroupFSTimelineStore to work in the absence of an RM
Date Fri, 19 Feb 2016 22:19:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154989#comment-15154989
] 

Steve Loughran commented on YARN-4696:
--------------------------------------

This is another iteration with some other things needed to debug and harden the code; probably
best to let me iterate on this until I've got my spark-> ATS work done, in case I find
more problems. As addressed in patch 005, the scan was having problems with 0 byte files.
This may be because I'm using local fs, not miniHDFS, this behaves differently.

I've also added more logging of what's going on, because there's enough async operations going
on in the client, it's a lot harder to debug why things aren't working: is it there's been
no flush client side, nothing picked up in the ATS, etc, etc.

> EntityGroupFSTimelineStore to work in the absence of an RM
> ----------------------------------------------------------
>
>                 Key: YARN-4696
>                 URL: https://issues.apache.org/jira/browse/YARN-4696
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: YARN-4696-001.patch, YARN-4696-002.patch, YARN-4696-003.patch, YARN-4696-005.patch
>
>
> {{EntityGroupFSTimelineStore}} now depends on an RM being up and running; the configuration
pointing to it. This is a new change, and impacts testing where you have historically been
able to test without an RM running.
> The sole purpose of the probe is to automatically determine if an app is running; it
falls back to "unknown" if not. If the RM connection was optional, the "unknown" codepath
could be called directly, relying on age of file as a metric of completion
> Options
> # add a flag to disable RM connect
> # skip automatically if RM not defined/set to 0.0.0.0
> # disable retries on yarn client IPC; if it fails, tag app as unknown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message