hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-13149) HBase MR is broken on Hadoop 2.5+ Yarn
Date Tue, 28 Apr 2015 19:00:08 GMT

     [ https://issues.apache.org/jira/browse/HBASE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nick Dimiduk updated HBASE-13149:
---------------------------------
    Release Note: 
In HBase 1.1.0 and above we have upgraded the version of Jackson dependencies (jackson-core-asl,
jackson-mapper-asl, jackson-jaxrs and jackson-xc) from 1.8.8 to 1.9.13. This is to follow
the upgrade to Jackson 1.9.13 in Hadoop 2.5 and above which causes Jackson class incompatibility
for HBase as reported in HBASE-13149.  Refer to HADOOP-10104 and YARN-2092 for additional
information. Jackson1.9.13 is not completely backward compatible with the prior version 1.8.8
used in HBase. See the Compatibility reports attached in HBASE-13149 and http://svn.codehaus.org/jackson/trunk/release-notes/VERSION
for more information.

This upgrade does not have direct impact on HBase users and HBase applications in most cases.
In the rare case where your HBase application uses Jackson directly AND your application has
compatibility issue with Jackson 1.9.13, you can do the following to mitigate the problem.

1. If you are on Hadoop 2.5 or above, and your HBase application involves running Yarn jobs,
we recommend you update your application to use Jackson 1.9.13. You may be able to explore
classpath isolation options (e.g. HADOOP-10893) or have your own classpath isolation strategy
that works for you, but the general recommendation is that you upgrade to Jackson 1.9.13.
2. You may choose to continue using Jackson 1.8.8 and not to use Jackson 1.9.13 in your classpath.
 You can also choose to replace the Jackson 1.9.13 jars in $HBASE_HOME/lib with 1.8.8 jars.
 It can work for you in the following cases:
a) You are on a Hadoop version earlier than Hadoop 2.5,  or
b) You are on Hadoop 2.5 or above, but your HBase application does not involve running Yarn
jobs.
3. You may experiment with further isolation using the shaded jars introduced with 1.1.0 via
HBASE-13517.

Note that it may not be tested or guaranteed that using Jackson 1.8.8 in $HBASE_HOME/lib will
work in future HBase releases.
It is recommended that your HBase application matches the Jackson version provided in HBase.

In HBase 0.98.x and HBase 1.0.x, we have NOT upgraded the version of Jackson dependencies.
If you are on Hadoop 2.5 or above, and your HBase application involves running Yarn jobs,
you may encounter Jackson class incomparability issue, as reported in HBASE-13149.

You can do the following to mitigate the problem:
1. Use 'hadoop jar' command to run your HBase jobs.
2. Explore classpath isolation options (e.g. HADOOP-10893) or have your own classpath isolation
strategy that works for you.
3. You can also choose to replace the Jackson 1.8.8 jars in $HBASE_HOME/lib with 1.9.13 jars
from your Hadoop lib directory. We have tested HBase 0.98 with Jackson 1.9.13.

  was:
In HBase 1.1.0 and above we have upgraded the version of Jackson dependencies (jackson-core-asl,
jackson-mapper-asl, jackson-jaxrs and jackson-xc) from 1.8.8 to 1.9.13.
This is to follow the upgrade to Jackson 1.9.13 in Hadoop 2.5 and above which causes Jackson
class incompatibility for HBase as reported in HBASE-13149. 
Refer to HADOOP-10104 and YARN-2092 for additional information.
Jackson1.9.13 is not completely backward compatible with the prior version 1.8.8 used in HBase.
See the Compatibility reports attached in HBASE-13149 and
http://svn.codehaus.org/jackson/trunk/release-notes/VERSION for more information.

This upgrade does not have direct impact on HBase users and HBase applications in most cases.
In the rare case where your HBase application uses Jackson directly AND your application has
compatibility issue with Jackson 1.9.13,
you can do the following to mitigate the problem.

1. If you are on Hadoop 2.5 or above, and your HBase application involves running Yarn jobs,
we recommend you update your application to use Jackson 1.9.13. You may be able to explore
classpath isolation options (e.g. HADOOP-10893) or have your own classpath isolation strategy
that works for you, but the general recommendation is that you upgrade to Jackson 1.9.13.
2. You may choose to continue using Jackson 1.8.8 and not to use Jackson 1.9.13 in your classpath.
 You can also choose to replace the Jackson 1.9.13 jars in $HBASE_HOME/lib with 1.8.8 jars.
 It can work for you in the following cases:
a) You are on a Hadoop version earlier than Hadoop 2.5,  or
b) You are on Hadoop 2.5 or above, but your HBase application does not involve running Yarn
jobs.

Note that it may not be tested or guaranteed that using Jackson 1.8.8 in $HBASE_HOME/lib will
work in future HBase releases.
It is recommended that your HBase application matches the Jackson version provided in HBase.

In HBase 0.98.x, we have NOT upgraded the version of Jackson dependencies. 
If you are on Hadoop 2.5 or above, and your HBase application involves running Yarn jobs,
you may encounter Jackson class incomparability issue, as reported in HBASE-13149.

You can do the following to mitigate the problem:
1. Use 'hadoop jar' command to run your HBase jobs.
2. Explore classpath isolation options (e.g. HADOOP-10893) or have your own classpath isolation
strategy that works for you.
3. You can also choose to replace the Jackson 1.8.8 jars in $HBASE_HOME/lib with 1.9.13 jars
from your Hadoop lib directory. We have tested HBase 0.98 with Jackson 1.9.13.


Really great note [~jinghe]! Let me add a quick bit about 1.0 and also HBASE-13517.

> HBase MR is broken on Hadoop 2.5+ Yarn
> --------------------------------------
>
>                 Key: HBASE-13149
>                 URL: https://issues.apache.org/jira/browse/HBASE-13149
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 1.0.0, 2.0.0, 0.98.10.1
>            Reporter: Jerry He
>            Assignee: Jerry He
>            Priority: Blocker
>             Fix For: 2.0.0, 1.1.0, 1.2.0
>
>         Attachments: HBASE-13149-0.98.patch, HBASE-13149-master.patch, jackson-core-asl-compat_report.html,
jackson-jaxrs-compat_report.html, jackson-mapper-asl-compat_report.html, jackson-xc-compat_report.html,
jackson_1.8_to_1.9_compat_report.html
>
>
> Running the server MR tools is not working on Yarn version 2.5+.
> Running org.apache.hadoop.hbase.mapreduce.Export:
> {noformat}
> Exception in thread "main" java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
>         at org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
>         at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:47)
>         at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:166)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.mapred.ResourceMgrDelegate.serviceInit(ResourceMgrDelegate.java:102)
>         at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>         at org.apache.hadoop.mapred.ResourceMgrDelegate.<init>(ResourceMgrDelegate.java:96)
>         at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
>         at org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
>         at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
>         at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
>         at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>         at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
>         at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
>         at org.apache.hadoop.hbase.mapreduce.Export.main(Export.java:189)
> {noformat}
> The problem seems to be the jackson jar version.  HADOOP-10104 updated jackson version
to 1.9.13.  YARN-2092 reported a problem as well.
> HBase is using jackson 1.8.8. This version of the jar in the classpath seem to cause
the problem.
> Should we upgrade to jackson 1.9.13? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message