hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-4696) EntityGroupFSTimelineStore to work in the absence of an RM
Date Wed, 24 Feb 2016 15:28:18 GMT

     [ https://issues.apache.org/jira/browse/YARN-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated YARN-4696:
    Attachment: YARN-4696-008.patch

Patch -008. This removes a subclass of RawLocalFileSystem that I'd been trying to instantiate
directly. That doesn't work...I won't go  into the details.

Note also that patch -007

# has the code to remember the cache option before the {{FileSystemTimelineWriter}} gets a
file, and restores it after
# has commented out the entire action of disabling the cache.

Why #2? It's to try to get a local FS with checksumming disabled picked up in test cases.
I've not got that working. 
Why #1? Because some other part of the JVM may want caching, and so they won't want this class
disabling it for them.

I'm assuming that the caching was disabled to ensure that if this class closed the fs instance
then the solution there is: don't close the FS when the service is stopped. We can rely on
Hadoop itself to stop all filesystems in JVM shutdown. Of course, if the concern is that its
other bits of code closing the FS, that's harder. In such a case, if I do manage to get my
local FS test working, then we may need a test-time option to not-disable the cache

> EntityGroupFSTimelineStore to work in the absence of an RM
> ----------------------------------------------------------
>                 Key: YARN-4696
>                 URL: https://issues.apache.org/jira/browse/YARN-4696
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: YARN-4696-001.patch, YARN-4696-002.patch, YARN-4696-003.patch, YARN-4696-005.patch,
YARN-4696-006.patch, YARN-4696-007.patch, YARN-4696-008.patch
> {{EntityGroupFSTimelineStore}} now depends on an RM being up and running; the configuration
pointing to it. This is a new change, and impacts testing where you have historically been
able to test without an RM running.
> The sole purpose of the probe is to automatically determine if an app is running; it
falls back to "unknown" if not. If the RM connection was optional, the "unknown" codepath
could be called directly, relying on age of file as a metric of completion
> Options
> # add a flag to disable RM connect
> # skip automatically if RM not defined/set to
> # disable retries on yarn client IPC; if it fails, tag app as unknown.

This message was sent by Atlassian JIRA

View raw message