hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Slusser <lslus...@gmail.com>
Subject Re: [Discuss] HBase with multiple threads
Date Fri, 06 Jun 2014 23:28:09 GMT
Thanks for that information, Jonatan, good to know about how the
connections worked.  Each of my threads create their own config objects so
i'm good to go there.


On Fri, Jun 6, 2014 at 4:19 PM, Jonathan Hsieh <jon@cloudera.com> wrote:

> this is subtle with the older API -- but to clarify:
>
> if you use the same config object you will share the same underlaying
> hconnection.  this does the multiplexing and demultiplexing so that each
> you only have one tcp connection per rs. 5 tables in five threads still
> only has on hconnection and one tcp connection per rs.
>
> if you create new config objects in each thread (like htablepool), you will
> end up with many hconnection objects and a new set of tcp connections per.
>   5 tables in five threads this way has 5 hconnection and ends up with 5
> tcp connections per rs.
>
> the new API makes this much more obvious.
>
> On Friday, June 6, 2014, Liam Slusser <lslusser@gmail.com> wrote:
>
> > What Jonathan said.  I create my threads and have each thread create
> their
> > own HTable connection.  So then there is no sharing - they all run
> > independent.  I use a shared queue to give each thread jobs.
> >
> > liam
> >
> >
> >
> > On Fri, Jun 6, 2014 at 3:57 PM, Jonathan Hsieh <jon@cloudera.com
> > <javascript:;>> wrote:
> >
> > > Hotec04,
> > >
> > > Create one one HConnection for the process and then create an HTable
> > > instance for each thread that uses the same HConnection. In newer hbase
> > it
> > > is HConnection.getTable(name), in older hbase it is new
> > > HTable(HConnection).
> > >
> > > HTablePool is deprecated, and shouldn't really be used.
> > >
> > > Jon.
> > >
> > >
> > > On Fri, Jun 6, 2014 at 3:01 PM, Hotec04 <hao.leng@dish.com
> > <javascript:;>> wrote:
> > >
> > > > Hi HBase users,
> > > >
> > > > I try to implement multiple threads for HBase writing process, but
> > looks
> > > > like the table.flushCommites() breaks thread easily. Then I add a
> > > > synchronized blocker for it, it flows but when I publish 50 messages,
> > > about
> > > > 2 of them are missing...I am using HTablePool as well, so when every
> > > thread
> > > > comes in, they grab one Htable but all Htables are using the same
> > table.
> > > >
> > > > Any ideas on how to work with HTable put operation for multiple
> > threads?
> > > >
> > > > Thank you!
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/Discuss-HBase-with-multiple-threads-tp4060081.html
> > > > Sent from the HBase User mailing list archive at Nabble.com.
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com <javascript:;> // @jmhsieh
> > >
> >
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message