hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi Ozawa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3874) Optimize and synchronize FS Reader and Writer Implementations
Date Thu, 03 Sep 2015 12:40:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728967#comment-14728967

Tsuyoshi Ozawa commented on YARN-3874:

[~djp], thanks for your reply.

Do you have any solid scenarios and cases that need this feature to work with optimization?

Yes, HDFS backing storage is useful for following scenarios(Quoting description of YARN-3841).

1. For Hadoop clusters which don't run HBase. LevelDB implementation in this case is not enough
since its durability is sometimes not enough for our customers.
2. For fallback from HBase when HBase cluster is temporary unavailable. Quoting ATS design
document of YARN-2928:
In the case the HBase
storage is not available, the plugin should buffer the writes temporarily (e.g. HDFS), and
them once the storage comes back online. Reading and writing to hdfs as the the backup storage
could potentially use the HDFS writer plugin unless the complexity of generalizing the HDFS
writer plugin for this purpose exceeds the benefits of reusing it here.

If current priority of this feature is low, I'd like to lead and support its development :-)

> Optimize and synchronize FS Reader and Writer Implementations
> -------------------------------------------------------------
>                 Key: YARN-3874
>                 URL: https://issues.apache.org/jira/browse/YARN-3874
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-3874-YARN-2928.01.patch, YARN-3874-YARN-2928.02.patch, YARN-3874-YARN-2928.03.patch
> Combine FS Reader and Writer Implementations and make them consistent with each other.

This message was sent by Atlassian JIRA

View raw message