hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Riccomini (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-896) Roll up for long lived YARN
Date Mon, 26 Aug 2013 17:39:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750288#comment-13750288
] 

Chris Riccomini commented on YARN-896:
--------------------------------------

bq. Because many of these systems roll their logs to avoid filling up disks we will probably
need a protocol of some sort for the container to communicate with the Node Manager when logs
are ready to be processed. 

Somewhat related, and a random thought: Samza is currently piping stdout and stderr to files
when it sets its AM and container commands (... 1>logs/stdout 2>logs/stderr). These
files never get rolled, and they can get very big for long-lived processes (10s-100s of gigs).
We encourage using a logging system (log4j) to handle this stuff, but there are cases where
the recommended use is not followed (sigh :)).

What do you guys think about tweaking the NM to consume the STDOUT/STDERR streams from the
ProcessBuilder when it executes AM/container commands, writing them to stdout/stderr files,
and rolling them periodically (e.g. daily)?
                
> Roll up for long lived YARN
> ---------------------------
>
>                 Key: YARN-896
>                 URL: https://issues.apache.org/jira/browse/YARN-896
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Robert Joseph Evans
>
> YARN is intended to be general purpose, but it is missing some features to be able to
truly support long lived applications and long lived containers.
> This ticket is intended to
>  # discuss what is needed to support long lived processes
>  # track the resulting JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message