hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
Date Mon, 05 Oct 2015 20:47:27 GMT

    [ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943999#comment-14943999

Sangjin Lee commented on YARN-4061:

Sorry it took me a while to get to this. Thanks for the proposal [~gtCarrera9]!

One potential area of concern is regarding flush and the associated contract with the client.
If the client wanted to write critical data synchronously and specifically wanted to block
until it receives a server (storage) response, how would that work in this scheme? The proposal
has the following for the flush situation:
When one log segment reaches a predefined size, or a timeĀ­ trigger, or an explicit flush
call happens, it is published to a log queue.

Since the actual storage writer (HBase) always acts on this queue asynchronously, it seems
that the client cannot have a synchronous write semantics. Is that a correct reading? If so,
how would we implement such a synchronous write?

> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>                 Key: YARN-4061
>                 URL: https://issues.apache.org/jira/browse/YARN-4061
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: FaulttolerantwriterforTimelinev2.pdf
> We need to build a timeline writer that can be resistant to backend storage down time
and timeline collector failures. 

This message was sent by Atlassian JIRA

View raw message