hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
Date Thu, 14 Mar 2013 21:00:14 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602729#comment-13602729

Nick Dimiduk commented on HBASE-7938:

bq. I guess you had an open issue for this, right?

Yeah, that's what I ment when I said "The test works as intended when run stand-alone, but
FAILS when run via maven. It looks like HFileOutputFormat is creating the paritions file on
localFs instead of HDFS." The test does spin up a cluster + mapred via IntegrationTestingUtility.
Scroll back in the logs and you'll see {{INFO mapreduce.HFileOutputFormat: Writing partition
information to}} and then a long local file path. That path ends up not existing when the
TotalOrderPartitioner gets around to looking for it; I don't know why. You'll also see an
error about "unable to create symlink" a little later on, because that file path no longer

This business is fixed in HBASE-4285. I do away with the symlink stuff and drop the partitions
file in /tmp instead of the working directory. I also request that the file be deleted at
FS close.

bq. Second question is that, whether we need to run the other 2 tests in TestImportTsv in
IT mode?

The point of the integration test (in my mind) is that it can be run on a real cluster. If
that's desirable for the tests in TestImportTsv, they might as well be merged. The alternative
is to try setting up TestImportTsv with a main that can be run w.o maven.

Personally, I don't think we gain very much by having the integration test stuff tied into
maven. We don't have a persistent cluster that Jenkins can run them against, so when they
run through maven they're using the minicluster, same as regular tests. After my experience
with this thing, I think we're better off writing them all as Tools with a main that can be
run against an externally specified conf.
> Add integration test for ImportTsv/LoadIncrementalHFiles workflow
> -----------------------------------------------------------------
>                 Key: HBASE-7938
>                 URL: https://issues.apache.org/jira/browse/HBASE-7938
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>             Fix For: 0.95.0, 0.98.0
>         Attachments: 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch,
0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch
> We have existing unit tests for smoke-testing the packaged MR jobs, however they do not
create a runtime environment that is true to running on a real MR cluster. This is particularly
true in regard to classpaths (HBASE-7934) but also other static state (HBASE-4802). An integration
test that can be pointed to run on a pseudo-distributed Hadoop deployed on localhost would
find these kinds of problems.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message