hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2229) ContainerId can overflow with RM restart
Date Tue, 22 Jul 2014 20:36:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14070849#comment-14070849

Siddharth Seth commented on YARN-2229:

I don't have enough context on the RM restart changes, but some comments.
- hashCode and equals are inconsistent in the latest patch. One uses getId(), the other uses
- Agree with Zhijie on not changing an @Stable method to @Unstable. Deprecate in this patch
itself ?
- ConverterUtils is a separate consideration. It is marked as @private - but is used in MapReduce
for example (and also in Tez). Looks like the toString method isn't being changed either,
whcih means to ConverterUtils method would continue to work.

Is the epoch change for internal use only ?, and only when being serialized via protocol buffers
? If epoch is only for internal consumption - I like [~ozawa]'s initial suggestion of introducing
a new field - which is not exposed to users, and is only used by versions of YARN which know
how to interpret it.

> ContainerId can overflow with RM restart
> ----------------------------------------
>                 Key: YARN-2229
>                 URL: https://issues.apache.org/jira/browse/YARN-2229
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2229.1.patch, YARN-2229.10.patch, YARN-2229.10.patch, YARN-2229.2.patch,
YARN-2229.2.patch, YARN-2229.3.patch, YARN-2229.4.patch, YARN-2229.5.patch, YARN-2229.6.patch,
YARN-2229.7.patch, YARN-2229.8.patch, YARN-2229.9.patch
> On YARN-2052, we changed containerId format: upper 10 bits are for epoch, lower 22 bits
are for sequence number of Ids. This is for preserving semantics of {{ContainerId#getId()}},
{{ContainerId#toString()}}, {{ContainerId#compareTo()}}, {{ContainerId#equals}}, and {{ConverterUtils#toContainerId}}.
One concern is epoch can overflow after RM restarts 1024 times.
> To avoid the problem, its better to make containerId long. We need to define the new
format of container Id with preserving backward compatibility on this JIRA.

This message was sent by Atlassian JIRA

View raw message