hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cristofer Weber <cristofer.we...@neogrid.com>
Subject RES: HBase and unit tests
Date Fri, 31 Aug 2012 12:35:19 GMT
Hi Ulrich,

Yes, I'm starting mini cluster inside @BeforeClass. There are 3 different test cases, and
between 2 and 15 tests per test case.


Best regards,

-----Mensagem original-----
De: ustaudinger@gmail.com [mailto:ustaudinger@gmail.com] Em nome de Ulrich Staudinger
Enviada em: sexta-feira, 31 de agosto de 2012 08:05
Para: user@hbase.apache.org
Assunto: Re: HBase and unit tests

Hi Cristofer,

> At least 15 seconds are spent on starting the mini cluster for each 
> test

and you are sure that you are reusing your mini cluster across unit tests?


On Fri, Aug 31, 2012 at 12:28 PM, Cristofer Weber < cristofer.weber@neogrid.com> wrote:

> Hi Sonal, Stack and Ulrich!
> Yes, I should provide more details :$
> I reached the links you provided when I was searching for a way to 
> start HBase with JUnit. From default, the only params I have changed 
> are Zookeeper port and the amount of nodes, which is 1 in my case. 
> Based on logs I suspect that most of time are spent with HDFS and 
> that's why I asked if there is a way to start a standalone instance of 
> HBase. The amount of data written at each test case would probably fit 
> in memstore anyway, and table cleansing between each test method is managed by a loop
of deletes.
> At least 15 seconds are spent on starting the mini cluster for each 
> test case.
> Right now I reminded that I should turn off WAL when running unit 
> tests :-), but this will not reflect on startup time.
> Thanks!!
> Best regards,
> Cristofer
> ________________________________________
> De: Ulrich Staudinger [ustaudinger@gmail.com]
> Enviado: sexta-feira, 31 de agosto de 2012 2:21
> Para: user@hbase.apache.org
> Assunto: Re: HBase and unit tests
> As a general advice, although you probably do take care of this, 
> instantiate the mini cluster only once in your junit test constructor 
> and not in every test method. at the end of each test, either cleanup 
> your hbase or use a different "area" per test.
> best regards,
> ulrich
> --
> connect on xing or linkedin. sent from my tablet.
> On 31.08.2012, at 06:46, Stack <stack@duboce.net> wrote:
> > On Thu, Aug 30, 2012 at 4:44 PM, Cristofer Weber 
> > <cristofer.weber@neogrid.com> wrote:
> >> Hi there!
> >>
> >> After I started studying HBase, I've searched for open source 
> >> projects
> backed by HBase and I found Titan distributed graph database (you 
> probably heard about it). As soon as I read in their documentation 
> that HBase adapter is experimental and suboptimal (disclaimer here:
> https://github.com/thinkaurelius/titan/wiki/Using-HBase) I volunteered 
> to help improving this adapter and since then I made a few changes to 
> improve on running tests (reduced from hours to minutes) and also an 
> improvement on search feature.
> >>
> >> Now I'm trying to break the dependency on a pre-installed HBase for
> unit tests and found miniCluster inside HBase tests, but minicluster 
> demands too much time to start and I don't know if tweaking on configs 
> will improve significantly. Is there a way to start a 'lightweight' 
> instance, like programatically starting a standalone instance?
> >>
> >
> > How much is 'too much time' Cristofer?  Do you want a standalone 
> > cluster
> at all?
> > St.Ack
> > P.S. If digging in this area, you might find the blog post by the 
> > sematextians of use:
> >
> http://blog.sematext.com/2010/08/30/hbase-case-study-using-hbasetestin
> gutility-for-local-testing-development/

Ulrich Staudinger

Connect online: https://www.xing.com/profile/Ulrich_Staudinger
View raw message