hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhihai xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3017) ContainerID in ResourceManager Log Has Slightly Different Format From AppAttemptID
Date Sun, 28 Jun 2015 03:46:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604503#comment-14604503
] 

zhihai xu commented on YARN-3017:
---------------------------------

I just found this change may cause problem in LogAggregation during rolling upgrade with NM-Recovery-supervised
enabled.
The following code in {{AggregatedLogFormat#getPendingLogFilesToUploadForThisContainer}} will
upload the log based on the containerId String. So we may miss uploading the old log files
after upgrade.
{code}
        File containerLogDir =
            new File(appLogDir, ConverterUtils.toString(this.containerId));
        if (!containerLogDir.isDirectory()) {
          continue; // ContainerDir may have been deleted by the user.
        }
        pendingUploadFiles
          .addAll(getPendingLogFilesToUpload(containerLogDir));
{code}
To support this issue, we also need make change in {{getPendingLogFilesToUploadForThisContainer}}
to compare containerId using {{ContainerId#fromString}}.
It looks like it makes sense to keep the old format for compatibility.

> ContainerID in ResourceManager Log Has Slightly Different Format From AppAttemptID
> ----------------------------------------------------------------------------------
>
>                 Key: YARN-3017
>                 URL: https://issues.apache.org/jira/browse/YARN-3017
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0
>            Reporter: MUFEED USMAN
>            Assignee: Mohammad Shahid Khan
>            Priority: Minor
>              Labels: PatchAvailable
>         Attachments: YARN-3017.patch, YARN-3017_1.patch, YARN-3017_2.patch, YARN-3017_3.patch
>
>
> Not sure if this should be filed as a bug or not.
> In the ResourceManager log in the events surrounding the creation of a new
> application attempt,
> ...
> ...
> 2014-11-14 17:45:37,258 INFO
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching
> masterappattempt_1412150883650_0001_000002
> ...
> ...
> The application attempt has the ID format "_1412150883650_0001_000002".
> Whereas the associated ContainerID goes by "_1412150883650_0001_02_".
> ...
> ...
> 2014-11-14 17:45:37,260 INFO
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up
> container Container: [ContainerId: container_1412150883650_0001_02_000001,
> NodeId: n67:55933, NodeHttpAddress: n67:8042, Resource: <memory:2048, vCores:1,
> disks:0.0>, Priority: 0, Token: Token { kind: ContainerToken, service:
> 10.10.70.67:55933 }, ] for AM appattempt_1412150883650_0001_000002
> ...
> ...
> Curious to know if this is kept like that for a reason. If not while using
> filtering tools to, say, grep events surrounding a specific attempt by the
> numeric ID part information may slip out during troubleshooting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message