hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joep Rottinghuis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
Date Fri, 16 Dec 2016 18:10:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755117#comment-15755117
] 

Joep Rottinghuis commented on YARN-4061:
----------------------------------------

For the next version of the patch I'll modify the code so that even if HBase is down we can
create a SpoolingBufferedMutatorImpl. I'll check a little whether we need to also push down
the connection creation into the code. Part of the issue there is that we now will be creating
multiple connections rather than having one shared one. Keep in mind that we have a BufferedMutator
per table. It would be unfortunate to create 5 connections.

wrt. "at least once" here is my reply from the Google doc:
bq. Yes, we guarantee at least once, if one considers spooling the mutation a delivery. The
code to replay those mutations is still missing. I'm not sure if that should belong here,
or if that should be a separate thing.

bq. From the Yarn timeline perspective we set the timestamp on each put explicitly exactly
for the reason of delayed mutations. We want to know when they happened, and not when they
were submitted. For our usecase, that makes the puts idempotent. Other use-cases may not need
this requirement, but they do need to deal with duplicate puts.

> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>
>                 Key: YARN-4061
>                 URL: https://issues.apache.org/jira/browse/YARN-4061
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Joep Rottinghuis
>              Labels: YARN-5355
>         Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage down time
and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message