hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3089) LinuxContainerExecutor does not handle file arguments to deleteAsUser
Date Thu, 05 Feb 2015 23:07:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308204#comment-14308204

Jason Lowe commented on YARN-3089:

bq. But if the user does not set any includePattern/excludePattern in LogAggregationContext,
and does not want to roll-over the logs, all container logs will be written into the files
with the same name(run the command, such as 1>>stdout, 2>>stderr ). After we aggregate
the logs into HDFS, the stdout/stderr will be deleted. In this case, this LRS app is affected.

This sounds pretty bad.  If the app doesn't roll its own logs but is normally not very chatty
such that it isn't much of an issue, we're going to blow away the apps logs every log-aggregation.roll-monitoring-interval-seconds
interval?  That interval isn't configurable per app, so I can see cases where someone puts
an app on the cluster that isn't super long running, doesn't roll its own logs, but then this
feature comes along and uploads a partial log and removes the active log files.  In other
words, fixing this bug may actually start deleting logs we shouldn't delete on clusters using
the LCE.  However that's a separate JIRA from this, as this is focusing on making the behavior
of the LCE consistent with the default executor wrt. deletes.

> LinuxContainerExecutor does not handle file arguments to deleteAsUser
> ---------------------------------------------------------------------
>                 Key: YARN-3089
>                 URL: https://issues.apache.org/jira/browse/YARN-3089
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Eric Payne
>            Priority: Blocker
>         Attachments: YARN-3089.v1.txt, YARN-3089.v2.txt, YARN-3089.v3.txt
> YARN-2468 added the deletion of individual logs that are aggregated, but this fails to
delete log files when the LCE is being used.  The LCE native executable assumes the paths
being passed are paths and the delete fails.

This message was sent by Atlassian JIRA

View raw message