Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Mon, 26 Aug 2013 17:39:58 +0000 (UTC)
From: "Chris Riccomini (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12655952.1372813766822.33337.1377538798242@arcas>
In-Reply-To: <JIRA.12655952.1372813766822@arcas>
References: <JIRA.12655952.1372813766822@arcas>
Subject: [jira] [Commented] (YARN-896) Roll up for long lived YARN
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750288#comment-13750288 ] 

Chris Riccomini commented on YARN-896:
--------------------------------------

bq. Because many of these systems roll their logs to avoid filling up disks we will probably need a protocol of some sort for the container to communicate with the Node Manager when logs are ready to be processed. 

Somewhat related, and a random thought: Samza is currently piping stdout and stderr to files when it sets its AM and container commands (... 1>logs/stdout 2>logs/stderr). These files never get rolled, and they can get very big for long-lived processes (10s-100s of gigs). We encourage using a logging system (log4j) to handle this stuff, but there are cases where the recommended use is not followed (sigh :)).

What do you guys think about tweaking the NM to consume the STDOUT/STDERR streams from the ProcessBuilder when it executes AM/container commands, writing them to stdout/stderr files, and rolling them periodically (e.g. daily)?
                
> Roll up for long lived YARN
> ---------------------------
>
>                 Key: YARN-896
>                 URL: https://issues.apache.org/jira/browse/YARN-896
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Robert Joseph Evans
>
> YARN is intended to be general purpose, but it is missing some features to be able to truly support long lived applications and long lived containers.
> This ticket is intended to
>  # discuss what is needed to support long lived processes
>  # track the resulting JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira