hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2
Date Mon, 05 Oct 2015 22:58:28 GMT

    [ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14944200#comment-14944200
] 

Sangjin Lee commented on YARN-4061:
-----------------------------------

I don't think the MR JHS is an apt comparison. First, we're dealing with a totally distributed
writer situation (individual jobs) for the MR JHS whereas the RM timeline collector would
be a single significant writer (again, it's the RM collector that I'm most worried about).
Also, JHS writes only a few large files (job conf, job history files, etc.), whereas the timeline
service will write a huge number of tiny writes. The volume of writes will be much larger
than the JHS use case.

Regarding the synchronous semantics, we really need to think it through. On the one hand,
we might consider handling the synchronous calls separate from the rest and outside the log
queue, but it's not clear how one can make it work alongside the asynchronous writes that
are going on.

> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>
>                 Key: YARN-4061
>                 URL: https://issues.apache.org/jira/browse/YARN-4061
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage down time
and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message