hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naveen" <naveen.moorj...@cleartrip.com>
Subject RE: Mass dumping of data has issues
Date Wed, 26 Sep 2012 07:09:49 GMT
Hi Ram,

I'm using HTablePool to get the instance for each thread. So, no, I'm not
sharing the same instance across multiple threads.

Regards,
Naveen

-----Original Message-----
From: Ramkrishna.S.Vasudevan [mailto:ramkrishna.vasudevan@huawei.com] 
Sent: Wednesday, September 26, 2012 12:15 PM
To: user@hbase.apache.org
Subject: RE: Mass dumping of data has issues

For the NPE that you got, is the same HTable instance shared by different
threads.  This is a common problem users encounter while using HTable across
multiple threads.
Pls check and ensure that the HTable is not shared.

Regards
Ram

> -----Original Message-----
> From: Naveen [mailto:naveen.moorjani@cleartrip.com]
> Sent: Wednesday, September 26, 2012 11:52 AM
> To: user@hbase.apache.org
> Subject: RE: Mass dumping of data has issues
> 
> Hi Dan,
> 
> I'm actually trying to simulate the kind of load we're expecting on 
> production servers(the intention is not to migrate data), hence a 
> self-written program over sqoop. (PS: I've actually tried sqoop)
> 
> Warm regards,
> Naveen
> 
> -----Original Message-----
> From: Dan Han [mailto:dannahan2008@gmail.com]
> Sent: Wednesday, September 26, 2012 7:20 AM
> To: user@hbase.apache.org
> Subject: Re: Mass dumping of data has issues
> 
> Hi Naveen,
> 
>    There is tool called Sqoop which supports importing the data from 
> relational database to HBase.
> https://blogs.apache.org/sqoop/entry/apache_sqoop_graduates_from_incub
> a
> tor
> 
> Maybe it can help you migrate the data easily.
> 
> Best Wishes
> Dan Han
> 
> On Mon, Sep 24, 2012 at 9:20 AM, Paul Mackles <pmackles@adobe.com>
> wrote:
> 
> > Did you adjust the writebuffer to a larger size and/or turn off 
> > autoFlush for the Htable? I've found that both of those settings can 
> > have a profound impact on write performance. You might also look at 
> > adjusting the handler count for the regionservers which by default 
> > is pretty low. You should also confirm that your splits are 
> > effective in
> distributing the writes.
> >
> > On 9/24/12 11:01 AM, "Naveen" <naveen.moorjani@cleartrip.com> wrote:
> >
> > >Hi,
> > >
> > >I've come across the following issue for which I'm unable to deduce 
> > >what the root-cause could be.
> > >
> > >Scenario:
> > >I'm trying to dump data(8.3M+ records) from mysql into a hbase 
> > >table using
> > >multi-threading(25 threads dumping 10 puts/tuples at a time).
> > >
> > >Config:
> > >hbase v 0.92.0
> > >hadoop v 1.0
> > >1 master + 4 slaves
> > >table is pre-split
> > >
> > >Issue:
> > >Getting a NPE because RPC call takes longer than timeout(default 60
> sec).
> > >I'm not worried about the NPE(it's been fixed in 0.92.1+ releases) 
> > >but about what could be causing RPC call to timeout on arbitrary 
> > >intervals.
> > >
> > >Custom printed log : pastebin.com/r85wv8Yt
> > >
> > >WARN [Thread-99255] (HConnectionManager.java:1587) - Failed all 
> > >from
> > >region=dump,a405cdd9-b5b7-4ec2-9f91-
> fea98d5cb656,1348331511473.77f13d
> > >455fd
> > >63
> > >c601816759b6ed575e8., hostname=hdslave1.company.com, port=60020
> > >java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> > >java.lang.NullPointerException
> > >       at
> > >java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > >       at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > >       at
> >
> >org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
> > >tatio
> > >n.
> > >processBatchCallback(HConnectionManager.java:1557)
> > >       at
> >
> >org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
> > >tatio
> > >n.
> > >processBatch(HConnectionManager.java:1409)
> > >       at
> > >org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900)
> > >       at
> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:777)
> > >       at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760)
> > >       at
> >
> >org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool
> > >.java
> > >:4
> > >02)
> > >       at coprocessor.dump.Dumper.run(Dumper.java:41)
> > >       at java.lang.Thread.run(Thread.java:662)
> > >
> > >Any help or insights are welcome.
> > >
> > >Warm Regards,
> > >Naveen
> > >
> >
> >



Mime
View raw message