hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1578) Fix how to handle ApplicationHistory about the container
Date Wed, 29 Jan 2014 19:46:10 GMT

    [ https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885709#comment-13885709
] 

Zhijie Shen commented on YARN-1578:
-----------------------------------

[~sinchii], thanks for your investigation. In my previous comment, I meant it should be fine
if the finish data of an container is not written by RM. Then, in this case, the finish data
should not exist in the persisted history file. Therefore, in the following code,
{code}
        if (entry.key.id.equals(containerId.toString())) {
          if (entry.key.suffix.equals(START_DATA_SUFFIX)) {
            ContainerStartData startData =
                parseContainerStartData(entry.value);
            mergeContainerHistoryData(historyData, startData);
            readStartData = true;
          } else if (entry.key.suffix.equals(FINISH_DATA_SUFFIX)) {
            ContainerFinishData finishData =
                parseContainerFinishData(entry.value);
            mergeContainerHistoryData(historyData, finishData);
            readFinishData = true;
          }
        }
{code}
The second inner condition is supposed be failed. However, it seems that the second inner
condition got passed, while the entry was actually not the byte[] to construct finish data
instance.

> Fix how to handle ApplicationHistory about the container
> --------------------------------------------------------
>
>                 Key: YARN-1578
>                 URL: https://issues.apache.org/jira/browse/YARN-1578
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: YARN-321
>            Reporter: Shinichi Yamashita
>            Assignee: Shinichi Yamashita
>         Attachments: YARN-1578.patch, application_1390978867235_0001, resoucemanager.log,
screenshot.png
>
>
> I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
> After the job end and when I accessed Web UI of HistoryServer, it displayed "500". And
HistoryServer daemon log was output as follows.
> {code}
> 2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error handling
URI: /applicationhistory/appattempt/appattempt_1389146249925_0008_000001
> java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> (snip...)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
>         at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
>         at org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
> (snip...)
> {code}
> I confirmed that there was container which was not finished from ApplicationHistory file.
> In ResourceManager daemon log, ResourceManager reserved this container, but did not allocate
it.
> Therefore, about a container which is not allocated, it is necessary to change how to
handle in ApplicationHistory.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message