hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Li Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3949) ensure timely flush of timeline writes
Date Thu, 23 Jul 2015 17:55:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639264#comment-14639264
] 

Li Lu commented on YARN-3949:
-----------------------------

Hi [~sjlee0], thanks for the patch! I think the patch LGTM in general. One comment is that,
in future we may want to distinguish if a writer is synchronous or asynchronous. Synchronous
writers will have their flush() methods as no-ops, but asynchronous writers will actually
flush the storage (but have the potential risk to lose data if there's no flush afterwards).
We may want to show the difference from their names? Anyway, since we only focus on HBase
and FS storage for now (I'll move and refactor Phoenix storage soon), I think we can proceed
with the current patch. 

> ensure timely flush of timeline writes
> --------------------------------------
>
>                 Key: YARN-3949
>                 URL: https://issues.apache.org/jira/browse/YARN-3949
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: YARN-3949-YARN-2928.001.patch
>
>
> Currently flushing of timeline writes is not really handled. For example, {{HBaseTimelineWriterImpl}}
relies on HBase's {{BufferedMutator}} to batch and write puts asynchronously. However, {{BufferedMutator}}
may not flush them to HBase unless the internal buffer fills up.
> We do need a flush functionality first to ensure that data are written in a reasonably
timely manner, and to be able to ensure some critical writes are done synchronously (e.g.
key lifecycle events).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message