hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3045) [Event producers] Implement NM writing container lifecycle events to ATS
Date Sat, 01 Aug 2015 00:07:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14650049#comment-14650049

Sangjin Lee commented on YARN-3045:

Sorry was not able to completely get it. Are you referring to TimelineClient's putEntities
& putEntitiesAsync ? or 
TimelineWriter's write & flush ? if its later, apart from RM most of the entities and
events of NM and application go through TimelineClient > TimelineCollector . So there should
be some way for NM and Client App(AM) to inform to flush the data right?

Sorry if my comments were not detailed enough. As you know, we have layers of APIs: {{TimelineClient}}
which clients (RM, NM, AM, ...) use, {{TimelineCollector}} that receives those calls and interacts
with the writer, and {{TimelineWriter}} that handles actual writes.

The point was that {{flush()}} belongs only in {{TimelineWriter}}. Along with that, it is
implicitly understood that {{TimelineWriter.write()}} may be asynchronous (i.e. may not write
the data to the storage synchronously or promptly). This API should be sufficient for {{TimelineCollector}}
to express synchronous/critical put operations and asynchronous/non-critical put operations.
{{TimelineCollector}} will *not* expose {{flush()}} directly to clients. Instead, it may use
things like {{putEntity()}} and {{putEntityAsync()}} to expose that semantics to the client.
In the simplest terms, {{TimelineCollector}} could implement {{putEntity()}} as {{putEntityAsync()}}
+ {{TimelineWriter.flush()}}. This is not the actual suggestion of the implementation, but
that would be an idea. We already have {{putEntity()}} and {{putEntityAsync()}}, but we haven't
yet used {{flush()}} to do this behavior.

And yes, we'll need another JIRA to differentiate {{putEntity()}} and {{putEntityAsync()}}
and use them at the right places. Currently, {{putEntity()}} does not call {{flush()}}, and
all client calls are using {{putEntity()}}.

But some of them related to localization i feel is related to ContainerEntity right? hope
the approach captured by me is fine ?

If these events are already associated with containers any way, they are not an issue, right?
I thought there were these events that are really application events, but specific to the
nodes? My comments were about those application events.

> [Event producers] Implement NM writing container lifecycle events to ATS
> ------------------------------------------------------------------------
>                 Key: YARN-3045
>                 URL: https://issues.apache.org/jira/browse/YARN-3045
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3045-YARN-2928.002.patch, YARN-3045-YARN-2928.003.patch, YARN-3045-YARN-2928.004.patch,
YARN-3045-YARN-2928.005.patch, YARN-3045-YARN-2928.006.patch, YARN-3045.20150420-1.patch
> Per design in YARN-2928, implement NM writing container lifecycle events and container
system metrics to ATS.

This message was sent by Atlassian JIRA

View raw message