hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ledion bitincka (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1440) Yarn aggregated logs are difficult for external tools to understand
Date Mon, 25 Nov 2013 19:56:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-1440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13831847#comment-13831847

ledion bitincka commented on YARN-1440:

[~zjffdu] - I stand corrected, there's currently one TFile per application per node with container_id
used as the key. While this is better than creating one file per container, it still leaves
the cluster exposed to the small file problem, imagine a 1000 node cluster, running 10000
apps/day - this would lead to 10M new TFiles. My hope is reduced complexity at the log file
level while punting the small file problem to the FS layer - the reasoning here being that
not all filesystems which can be used on Hadoop have a small file problem!

> Yarn aggregated logs are difficult for external tools to understand
> -------------------------------------------------------------------
>                 Key: YARN-1440
>                 URL: https://issues.apache.org/jira/browse/YARN-1440
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: ledion bitincka
>              Labels: log-aggregation, logs, tfile, yarn
> The log aggregation feature in Yarn is awesome! However, the file type and format in
which the log files are aggregated into (TFile) should either be much simpler or be made pluggable.
The current TFile format forces anyone who wants to see the files to either 
> a) use the web UI
> b) use the CLI tools (yarn logs)  or 
> c) write custom code to read the files 
> My suggestion would be to simplify the log collection by collecting and writing the raw
log files into a directory structure as follows: 
> {noformat}
> /{log-collection-dir}/{app-id}/{container-id}/{log-file-name} 
> {noformat}
> This way the application developers can (re)use a much wider array of tools to process
the logs. 
> For the readers who are not familiar with logs and their format you can find more info
the following two blog posts:
> http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
> http://blogs.splunk.com/2013/11/18/hadoop-2-0-rant/

This message was sent by Atlassian JIRA

View raw message