hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-975) Add a file-system implementation for history-storage
Date Mon, 21 Oct 2013 20:49:44 GMT

    [ https://issues.apache.org/jira/browse/YARN-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13801062#comment-13801062

Zhijie Shen commented on YARN-975:

As to the scalability of the implementation, it's obvious that 1 application will result in
1 history file.

Did some local test of 1 application, 1 application attempt and multiple containers. The sizes
of the file area:

| \# of containers | 100 | 1K | 10K | 100K |
| size of the file | 19KB | 184KB | 1.8MB | 19MB |

The size of the file is proportional to the number of containers. Even there're 100K containers
of one the application, the file size seems to be still acceptable. Of course, the file size
is also determined by the variable-length fields. For example, if every history data contains
a pretty long logURL, the file size will be certainly larger. However, the size should be
on the same order of magnitude.

> Add a file-system implementation for history-storage
> ----------------------------------------------------
>                 Key: YARN-975
>                 URL: https://issues.apache.org/jira/browse/YARN-975
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>         Attachments: YARN-975.1.patch, YARN-975.2.patch, YARN-975.3.patch, YARN-975.4.patch,
YARN-975.5.patch, YARN-975.6.patch
> HDFS implementation should be a standard persistence strategy of history storage

This message was sent by Atlassian JIRA

View raw message