hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2517) Implement TimelineClientAsync
Date Tue, 25 Nov 2014 01:49:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223923#comment-14223923

Zhijie Shen commented on YARN-2517:

Thanks for sharing your idea, Hitesh, Mit and Sangjin! I'd like to make some clarification.
The error that the handler wants to take care of is not the communication problem, but the
problem that happens when the server is processing the posted timeline entity (See TimelinePutResponse).
It could be the data integrity issue that is messed up by the app. For example, the posted
Entity A in Domain 1 is trying to relate to Entity B in Domain 2. It's fine if in some use
cases, the app doesn't want to care about it, and consequently doesn't require an Ack. The
app can just go ahead without providing the handler. However, it's still better to be generic
enough to cover the other use cases where the app wants to make sure it has the timeline data
persisted, or at least know it is succeeded or not.

Queueing/messaging layer may help to mitigate the communication problem, but aforementioned
data problem still could happen, such that the app may still want to hear about the put response.
However, it sounds right that the implementation of this layer will affect that of async call.
It makes sense to wait until we make it clear how to make client <-> TS communication
reliable. If we eventually find handler in async call is difficult and will further prevent
optimization, "a sync write (as it stands now) for critical data and an async write which
is basically fire-and-forge" sounds a reasonable alternative plan.

> Implement TimelineClientAsync
> -----------------------------
>                 Key: YARN-2517
>                 URL: https://issues.apache.org/jira/browse/YARN-2517
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Zhijie Shen
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2517.1.patch, YARN-2517.2.patch
> In some scenarios, we'd like to put timeline entities in another thread no to block the
current one.
> It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can
buffer entities, put them in a separate thread, and have callback to handle the responses.

This message was sent by Atlassian JIRA

View raw message