hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2468) Log handling for LRS
Date Fri, 03 Oct 2014 02:05:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157588#comment-14157588
] 

Xuan Gong commented on YARN-2468:
---------------------------------

bq. Also ContainerLogAggregator.uploadedFileMeta is also not needed to be a class member.

I think ContainerLogAggregator.uploadedFileMeta is needed to be a class member. It is used
to keep track of all previous uploaded log files for each container. We will use this information
to decide whether this log can be aggregated. 

New patch addressed other comments

> Log handling for LRS
> --------------------
>
>                 Key: YARN-2468
>                 URL: https://issues.apache.org/jira/browse/YARN-2468
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-2468.1.patch, YARN-2468.10.patch, YARN-2468.11.patch, YARN-2468.2.patch,
YARN-2468.3.patch, YARN-2468.3.rebase.2.patch, YARN-2468.3.rebase.patch, YARN-2468.4.1.patch,
YARN-2468.4.patch, YARN-2468.5.1.patch, YARN-2468.5.1.patch, YARN-2468.5.2.patch, YARN-2468.5.3.patch,
YARN-2468.5.4.patch, YARN-2468.5.patch, YARN-2468.6.1.patch, YARN-2468.6.patch, YARN-2468.7.1.patch,
YARN-2468.7.patch, YARN-2468.8.patch, YARN-2468.9.1.patch, YARN-2468.9.patch
>
>
> Currently, when application is finished, NM will start to do the log aggregation. But
for Long running service applications, this is not ideal. The problems we have are:
> 1) LRS applications are expected to run for a long time (weeks, months).
> 2) Currently, all the container logs (from one NM) will be written into a single file.
The files could become larger and larger.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message