hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2517) Implement TimelineClientAsync
Date Mon, 24 Nov 2014 19:00:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223299#comment-14223299

Sangjin Lee commented on YARN-2517:

It might be conceptually cleaner (and simpler to the clients) to have the clients post the
events but not deal with what would happen if the result was not successfully sent. Even the
very concept of whether the result was "sent" is problematic. An ATS writer implementation
could make a decision of buffering a certain amount of data before it physically writes it
to the backing storage (as optimization).

If the client needs to know whether the event was truly sent to the backing storage and also
deal with its failure, it may lead to ATS writer implementations leaking to clients and may
limit the way an ATS write implementation can be optimized, etc.

How about a sync write (as it stands now) for critical data and an async write which is basically
fire-and-forget? The understanding there would be the async write is basically a best effort
and it would fall on the implementation of ATS writes to try to deliver the events to the
backing storage as reliably and optimally as it can (but in theory no guarantees still). Then
the async write can even be enabled with a boolean flag.

> Implement TimelineClientAsync
> -----------------------------
>                 Key: YARN-2517
>                 URL: https://issues.apache.org/jira/browse/YARN-2517
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Zhijie Shen
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2517.1.patch, YARN-2517.2.patch
> In some scenarios, we'd like to put timeline entities in another thread no to block the
current one.
> It's good to have a TimelineClientAsync like AMRMClientAsync and NMClientAsync. It can
buffer entities, put them in a separate thread, and have callback to handle the responses.

This message was sent by Atlassian JIRA

View raw message