hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sushanth Sowmyan (JIRA)" <>
Subject [jira] [Commented] (HIVE-4388) HBase tests fail against Hadoop 2
Date Tue, 10 Sep 2013 18:09:55 GMT


Sushanth Sowmyan commented on HIVE-4388:

Hi Brock,

>From looking through the patch, I'm +1 with your basic intent.

Your patch does not apply cleanly with HEAD, but that's largely because of the hcatalog refactoring
done recently with HIVE-4869, HIVE-5163 and HIVE-5197. Going back to d0da259d02b546979b93a411493b1704f488b72d
allows me to test most of these (although still has a problem patching build.xml, but I was
able to resolve that manually) But I wanted to put in a tentative review before you put in
the effort to rebase it. As of right now, I think it looks good to rebase and bring this patch
up to date.


Additional thoughts:

I did, however, have had similar experiences trying to make hbase tests work with hadoop 2.x,
and faced a couple of further problems. When fetching from maven-ant-tasks, I noticed that
hbase was still bringing in hadoop1 transitive dependencies. This doesn't hurt our unit tests,
but when running HCatalog E2E tests, caused a couple of situations where the presence of the
hadoop1 client jar in hive lib dir would cause hbase tests to start failing. To fix this,
we had to set a system property called hadoop.profile before it fetched the correct transitive
dependencies. (This is different from the mvn.hadoop.profile = "hadoop23" that hive and hcat
use.) (This was, however, 2 weeks back, and I see a couple of pom.xml changes since then,
so I'm unsure if the hbase pom.xml has changed to make hadoop 2.x their default profile, and
if so, that will fix this, but if so, will break it for hadoop 1.x E2E tests - we need to
set the appropriate system property to be sure.) I will further test this today to see if
we still need it. (Another case where we'd be okay if we used mvn, but in the maven-ant-tasks
world, we need to do a bit of weird gluing.)

Also, one of the comments in your patch (to hcatalog/build.xml) about setting a property build.dir.hive
and triggering if not building from top level caught my eye. I do not recommend building hcatalog
by itself, outside of being called from hive's build.xml. If need be, for testing, for eg.,
overriding -Diterate.hive.full.tests from the root level works better. Having dealt with build
systems that try to build hive and hcatalog as separate components post-merge, I can vouch
that it's very painful, and some build fixes get very kludgy very fast. I'm not against setting
the build.dir property itself - it makes the build more robust, but I'm not sure that promoting
that style of usage is something we should do.

> HBase tests fail against Hadoop 2
> ---------------------------------
>                 Key: HIVE-4388
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: HBase Handler
>            Reporter: Gunther Hagleitner
>            Assignee: Brock Noland
>         Attachments: HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch,
HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388.patch, HIVE-4388-wip.txt
> Currently we're building by default against 0.92. When you run against hadoop 2 (
builds fail because of: HBASE-5963.
> HIVE-3861 upgrades the version of hbase used. This will get you past the problem in HBASE-5963
(which was fixed in 0.94.1) but fails with: HBASE-6396.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message