hive-gitbox mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hive] stoty opened a new pull request #2259: [WIP] HIVE-25101: Remove HBase libraries from HBase distribution
Date Tue, 11 May 2021 10:36:17 GMT

stoty opened a new pull request #2259:
URL: https://github.com/apache/hive/pull/2259


   ### What changes were proposed in this pull request?
   - Mark all HBase dependencies as provided, so that they are not include in the distribution
package
   - Add the hbase mapredcp jars to AUX_PARAM in the hive startup script
   - Remove the code that adds the hbase dependencies to the MR/TEZ jobs automatically
   - Let HiveSparkClientFactory work even if it cannot access the HBase libraries
   
   ### Why are the changes needed?
   Hive currently includes the HBase libraries in its distribution package, and also adds
the HBase libraries separately 
   vi the 'hbase mapredcp' command in the startup script.
   
   There are multiple problems with this:
   - It's redundant
   - The included HBase libraries are ancient, and cannot be easily updated 
   - The hbase-shaded-mapredcp.jar added by hbase mapredcp is incompatible with the included
unshaded libraries on an API level, and the ordering of the classes, and thus the effective
api can be different between the local and the MR/Tez classpath
   
   ### Does this PR introduce _any_ user-facing change?
   With this change, if the HBASE_HOME environment variable is not set, then the HBase libraries
will not
   be available in Hive. This is change from the current behaviour.
   Also, with this change there is no need to manually replace the HBase libraries in the
Hive distribution.
   
   ### How was this patch tested?
   I have built a local pseudodistributed cluster, and checked that the hbase handler works
with these changes.
   AFAIK Hive does not have end-to-end tests that can test the generated package and start-up
scripts.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Mime
View raw message