hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
Date Mon, 05 Oct 2015 19:05:28 GMT

    [ https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943849#comment-14943849

Naganarasimha G R commented on YARN-3367:

Thanks for the feed back [~sjlee0],
bq. If the client is fine without the server response, it clearly implies flush is not needed

Yes i agree with this, but what should be the behavior of Sync calls ? IMO in the wake of
YARN-4061  (Fault tolerant writer for timeline v2), we need not worry abt it either, Thoughts
bq. First of all, the timestamp should be set for most of the entities, metrics, events, etc.,
and the server should rely on the timestamps to resolve ordering. 
Well i can understand if all the events are received and timestamp is filled at the client
side we need not worry abt the order but in the case client goes down send some events out
of order ? like containerFinished event gets Published but logAggregation does not succeed.
And from Client App side not sure how important is the order to be maintained.

bq.   I don't see a whole lot of situations where the timeline client can do this easily and
Well the approach i thought is to block publishing further entities/events through sync /async
calls till the events in Timeline client queue is cleared. But i don't completely see the
need for them until its very necessary to maintain the order. [~djp] can you please comment
on this part as in this jira description you have targetted to get the events in order.

> Replace starting a separate thread for post entity with event loop in TimelineClient
> ------------------------------------------------------------------------------------
>                 Key: YARN-3367
>                 URL: https://issues.apache.org/jira/browse/YARN-3367
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Junping Du
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3367.YARN-2928.001.patch
> Since YARN-3039, we add loop in TimelineClient to wait for collectorServiceAddress ready
before posting any entity. In consumer of  TimelineClient (like AM), we are starting a new
thread for each call to get rid of potential deadlock in main thread. This way has at least
3 major defects:
> 1. The consumer need some additional code to wrap a thread before calling putEntities()
in TimelineClient.
> 2. It cost many thread resources which is unnecessary.
> 3. The sequence of events could be out of order because each posting operation thread
get out of waiting loop randomly.
> We should have something like event loop in TimelineClient side, putEntities() only put
related entities into a queue of entities and a separated thread handle to deliver entities
in queue to collector via REST call.

This message was sent by Atlassian JIRA

View raw message