hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3089) LinuxContainerExecutor does not handle file arguments to deleteAsUser
Date Wed, 04 Feb 2015 23:36:36 GMT

    [ https://issues.apache.org/jira/browse/YARN-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306220#comment-14306220
] 

Xuan Gong commented on YARN-3089:
---------------------------------

bq. The call of DeletionService.delete where basepaths are specified but subdir is empty/null
is new with YARN-2468, specifically in this part of the code in AppLogAggregatorImpl.uploadLogsForContainers:

Thanks for explanation. Jason. 

bq. do you still want this committed to 2.6.1 given it should only affect LRS apps?

For normal applications, the application logs directory will be deleted after the application
is finished. As Jason mentioned, this action gets succeed.  
For the LRS applications, We deleted the logs which is already uploaded. 
{code}
this.delService.delete(this.userUgi.getShortUserName(), null,
          uploadedFilePathsInThisCycle
            .toArray(new Path[uploadedFilePathsInThisCycle.size()]));
{code}

So, whether it can affect the LRS apps, my answer is "depends on". It is users' responsibility
to roll over their logs. YARN will upload and clean the logs which exist in the log directory.
Currently, it depends on how users want to roll-over their application logs and set up the
LogAggregationContext. 
For example, If the users roll-over the logs with some pattern, say "log.1, log.2, ..., etc",
and set up the includePattern in LogAggregationContext with something like "log.(\\d+)". So,
YARN will only upload the logs (log.1, log.2), and delete the logs (log.1, log.2) afterward.
In this case, *i do not think this LRS app is affected at all*.

But if the user does not set any includePattern/excludePattern in  LogAggregationContext,
and does not want to roll-over the logs, all container logs will be written into the files
with the same name(run the command, such as 1>>stdout, 2>>stderr ). After we aggregate
the logs into HDFS, the stdout/stderr will be deleted. *In this case, this LRS app is affected.*

> LinuxContainerExecutor does not handle file arguments to deleteAsUser
> ---------------------------------------------------------------------
>
>                 Key: YARN-3089
>                 URL: https://issues.apache.org/jira/browse/YARN-3089
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Eric Payne
>            Priority: Blocker
>         Attachments: YARN-3089.v1.txt, YARN-3089.v2.txt, YARN-3089.v3.txt
>
>
> YARN-2468 added the deletion of individual logs that are aggregated, but this fails to
delete log files when the LCE is being used.  The LCE native executable assumes the paths
being passed are paths and the delete fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message