hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3874) Combine FS Reader and Writer Implementations
Date Tue, 21 Jul 2015 13:39:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635115#comment-14635115

Varun Saxena commented on YARN-3874:

[~sjlee0], [~zjshen], kindly review.

Following has been done :
# Both FS reader and writer implementations have been made consistent with each other. Classes
have not yet been combined as no final decision was taken on it in YARN-3051. We can decide
if we will combine the classes or not.
# Moved some of the code common to reader and writer impls into {{TimelineStorageUtils}}
# Writer impl will now write app flow mapping file.
# As [~zjshen] said in YARN-3051, used {{FileSystem}} class so that it can be used for HDFS
as well.
# Entity files would now have created time in the file name. This will be used to limit the
entities returned and do filtering based on created time.
# Modified times as of now, wont be part of file name as we will have to consistently change
filenames if we do that. If required, that can be included as well. We can use rename operation.
Thoughts ?
# Writer would choose the file based on entity ID if created time is not given in request.
If created time is 0 or negative and file for an entity id is being written for the first
time, an error will be sent back.
# App flow mapping will be cached in reader and writer on start. And entries will be added
in reader whenever mapping has to be queried from file

> Combine FS Reader and Writer Implementations
> --------------------------------------------
>                 Key: YARN-3874
>                 URL: https://issues.apache.org/jira/browse/YARN-3874
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: YARN-2928
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-3874-YARN-2928.01.patch
> Combine FS Reader and Writer Implementations and make them consistent with each other.

This message was sent by Atlassian JIRA

View raw message