hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cristofer Weber <cristofer.we...@neogrid.com>
Subject RES: HBase and unit tests
Date Fri, 31 Aug 2012 10:28:59 GMT
Hi Sonal, Stack and Ulrich!

Yes, I should provide more details :$

I reached the links you provided when I was searching for a way to start HBase with JUnit.
From default, the only params I have changed are Zookeeper port and the amount of nodes, which
is 1 in my case. Based on logs I suspect that most of time are spent with HDFS and that's
why I asked if there is a way to start a standalone instance of HBase. The amount of data
written at each test case would probably fit in memstore anyway, and table cleansing between
each test method is managed by a loop of deletes.

At least 15 seconds are spent on starting the mini cluster for each test case. 

Right now I reminded that I should turn off WAL when running unit tests :-), but this will
not reflect on startup time.


Best regards,

De: Ulrich Staudinger [ustaudinger@gmail.com]
Enviado: sexta-feira, 31 de agosto de 2012 2:21
Para: user@hbase.apache.org
Assunto: Re: HBase and unit tests

As a general advice, although you probably do take care of this,
instantiate the mini cluster only once in your junit test constructor
and not in every test method. at the end of each test, either cleanup
your hbase or use a different "area" per test.

best regards,

connect on xing or linkedin. sent from my tablet.

On 31.08.2012, at 06:46, Stack <stack@duboce.net> wrote:

> On Thu, Aug 30, 2012 at 4:44 PM, Cristofer Weber
> <cristofer.weber@neogrid.com> wrote:
>> Hi there!
>> After I started studying HBase, I've searched for open source projects backed by
HBase and I found Titan distributed graph database (you probably heard about it). As soon
as I read in their documentation that HBase adapter is experimental and suboptimal (disclaimer
here: https://github.com/thinkaurelius/titan/wiki/Using-HBase) I volunteered to help improving
this adapter and since then I made a few changes to improve on running tests (reduced from
hours to minutes) and also an improvement on search feature.
>> Now I'm trying to break the dependency on a pre-installed HBase for unit tests and
found miniCluster inside HBase tests, but minicluster demands too much time to start and I
don't know if tweaking on configs will improve significantly. Is there a way to start a 'lightweight'
instance, like programatically starting a standalone instance?
> How much is 'too much time' Cristofer?  Do you want a standalone cluster at all?
> St.Ack
> P.S. If digging in this area, you might find the blog post by the
> sematextians of use:
> http://blog.sematext.com/2010/08/30/hbase-case-study-using-hbasetestingutility-for-local-testing-development/

View raw message