hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
Date Thu, 14 Mar 2013 21:00:14 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602729#comment-13602729
] 

Nick Dimiduk commented on HBASE-7938:
-------------------------------------

bq. I guess you had an open issue for this, right?

Yeah, that's what I ment when I said "The test works as intended when run stand-alone, but
FAILS when run via maven. It looks like HFileOutputFormat is creating the paritions file on
localFs instead of HDFS." The test does spin up a cluster + mapred via IntegrationTestingUtility.
Scroll back in the logs and you'll see {{INFO mapreduce.HFileOutputFormat: Writing partition
information to}} and then a long local file path. That path ends up not existing when the
TotalOrderPartitioner gets around to looking for it; I don't know why. You'll also see an
error about "unable to create symlink" a little later on, because that file path no longer
exists.

This business is fixed in HBASE-4285. I do away with the symlink stuff and drop the partitions
file in /tmp instead of the working directory. I also request that the file be deleted at
FS close.

bq. Second question is that, whether we need to run the other 2 tests in TestImportTsv in
IT mode?

The point of the integration test (in my mind) is that it can be run on a real cluster. If
that's desirable for the tests in TestImportTsv, they might as well be merged. The alternative
is to try setting up TestImportTsv with a main that can be run w.o maven.

Personally, I don't think we gain very much by having the integration test stuff tied into
maven. We don't have a persistent cluster that Jenkins can run them against, so when they
run through maven they're using the minicluster, same as regular tests. After my experience
with this thing, I think we're better off writing them all as Tools with a main that can be
run against an externally specified conf.
                
> Add integration test for ImportTsv/LoadIncrementalHFiles workflow
> -----------------------------------------------------------------
>
>                 Key: HBASE-7938
>                 URL: https://issues.apache.org/jira/browse/HBASE-7938
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>             Fix For: 0.95.0, 0.98.0
>
>         Attachments: 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch,
0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch
>
>
> We have existing unit tests for smoke-testing the packaged MR jobs, however they do not
create a runtime environment that is true to running on a real MR cluster. This is particularly
true in regard to classpaths (HBASE-7934) but also other static state (HBASE-4802). An integration
test that can be pointed to run on a pseudo-distributed Hadoop deployed on localhost would
find these kinds of problems.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message