hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5193) For long running services, aggregate logs when a container completes instead of when the app completes
Date Thu, 02 Jun 2016 18:20:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312822#comment-15312822

Siddharth Seth commented on YARN-5193:

Log rolling should help. I'm yet to try it out. Do you happen to know how it works when a
container dies - will the logs be aggregated immediately, or after the time window.

bq. Main thing to watch out for here is additional load to the namenode.
Yes. The original change to aggregate at the end was required for shorter running jobs with
more container churn. For a longer running service - containers will likely not go down very
often, and it should be oK to upload logs occasionally (without keeping connections open).

> For long running services, aggregate logs when a container completes instead of when
the app completes
> ------------------------------------------------------------------------------------------------------
>                 Key: YARN-5193
>                 URL: https://issues.apache.org/jira/browse/YARN-5193
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
> For a long running service, containers will typically not complete very often. However,
when a container completes - it would be useful to aggregate the logs right then, instead
of waiting for the app to complete.
> This will allow the command line log tool to lookup containers for an app from the log
file index itself, instead of having to go and talk to YARN. Talking to YARN really only works
if ATS is enabled, and YARN is configured to publish container information to ATS (That may
not always be the case - since this can overload ATS quite fast).
> There's some added benefits like cleaning out local disk space early, instead of waiting
till the app completes. (There's probably a separate jira somewhere about cleanup of container
for long running services anyway)
> cc [~vinodkv], [~xgong]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message