hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6354) RM fails to upgrade to 2.8 with leveldb state store
Date Thu, 16 Mar 2017 20:34:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928843#comment-15928843
] 

Jason Lowe commented on YARN-6354:
----------------------------------

Sample stacktrace:
{noformat}
2017-03-16 15:17:26,616 INFO  [main] service.AbstractService (AbstractService.java:noteFailure(272))
- Service ResourceManager failed in state STARTED; cause: java.lang.StringIndexOutOfBoundsException:
String index out of range: -17
java.lang.StringIndexOutOfBoundsException: String index out of range: -17
	at java.lang.String.substring(String.java:1931)
	at org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore.loadReservationState(LeveldbRMStateStore.java:289)
	at org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore.loadState(LeveldbRMStateStore.java:274)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:690)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1097)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1137)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1133)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1133)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1173)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1338)
{noformat}

This was broken by YARN-3736.  The recovery code is seeking to the RM_RESERVATION_KEY_PREFIX
but failing to verify that the keys it sees in the loop actually have that key prefix.  Here's
the relevant code:
{code}
      iter = new LeveldbIterator(db);
      iter.seek(bytes(RM_RESERVATION_KEY_PREFIX));
      while (iter.hasNext()) {
        Entry<byte[],byte[]> entry = iter.next();
        String key = asString(entry.getKey());

        String planReservationString =
            key.substring(RM_RESERVATION_KEY_PREFIX.length());
        String[] parts = planReservationString.split(SEPARATOR);
        if (parts.length != 2) {
          LOG.warn("Incorrect reservation state key " + key);
          continue;
        }
{code}

The only way to terminate this loop is when the iterator runs out of keys, therefore the iteration
loop will scan through *all* the keys in the database starting at the reservation key to the
end.  If any key encountered is too short then we'll get the out of bounds exception when
we try to do the substring.  

Pinging [~adhoot] and [~asuresh] who were involved in YARN-3736.

> RM fails to upgrade to 2.8 with leveldb state store
> ---------------------------------------------------
>
>                 Key: YARN-6354
>                 URL: https://issues.apache.org/jira/browse/YARN-6354
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.8.0
>            Reporter: Jason Lowe
>            Priority: Critical
>
> When trying to upgrade an RM to 2.8 it fails with a StringIndexOutOfBoundsException trying
to load reservation state.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message