hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Gray" <jl...@streamy.com>
Subject RE: Split META manually
Date Fri, 12 Mar 2010 10:02:41 GMT
Are these concurrent threads running in a single JVM?  Or is this mapreduce
and you run 600 tasks at once?

How big is your cluster and what version of HBase are you on?

> -----Original Message-----
> From: y_823910@tsmc.com [mailto:y_823910@tsmc.com]
> Sent: Friday, March 12, 2010 1:42 AM
> To: hbase-user@hadoop.apache.org
> Subject: RE: Split META manually
> 
> 
> Are you seeing a META bottleneck even after clients are running for
> some
> time? I think its right!
> 
> My job total count is 600.(my job is direct read hbase data and keep
> there
> in local machine for join)
> then write the joining result back to hbase.
> If I start 600 concurrent program, it took 2329 sec.
> (600 clients visit META concurrently and aggressively)
> If I just start 100 concurrent program then keep the META info for the
> subsequential 5 round of job.
> (only 100 concurrent program will visit META in the first round), it
> took
> 839 sec.
> So I think that you said that hbase can't sustain so many concurrent
> META
> requests is right.
> 
> 
> 
> Fleming Chiu(邱宏明)
> 707-6128
> y_823910@tsmc.com
> 週一無肉日吃素救地球(Meat Free Monday Taiwan)
> 
> 
> 
> 
> 
>                       "Jonathan Gray"
>                       <jlist@streamy.co        To:      <hbase-
> user@hadoop.apache.org>
>                       m>                       cc:      (bcc:
> Y_823910/TSMC)
>                                                Subject: RE: Split META
> manually
>                       2010/03/12 03:11
>                       PM
>                       Please respond to
>                       hbase-user
> 
> 
> 
> 
> 
> 
> Fleming,
> 
> We're looking at a few different ideas for this problem right now.
> 
> One is to make an efficient method for warming up a clients META cache
> by
> issuing a META scan for a single table or all tables.  This will be
> significantly faster than lots of gets.
> 
> The other bigger change is that META may move (at least partially) into
> ZooKeeper for 0.21.  This would be beneficial as ZK allows for reading
> from
> many replicas.
> 
> Splitting META sounds like a good idea but as of today it's really not
> well
> supported by HBase.  I think most of the team is working towards the
> above
> changes before making a split META work.  If you're really interested
> in
> it,
> coming up with some good unit tests is a good start and maybe we can
> work
> on
> that too.
> 
> Are you seeing a META bottleneck even after clients are running for
> some
> time?  It sounds like you just did a heavy concurrency test of
> artificial
> META reads.  The clients aggressively cache META results so they should
> not
> sustain their META requests after warmed.
> 
> JG
> 
> > -----Original Message-----
> > From: y_823910@tsmc.com [mailto:y_823910@tsmc.com]
> > Sent: Thursday, March 11, 2010 10:35 PM
> > To: hbase-user@hadoop.apache.org
> > Subject: Re: Split META manually
> >
> > It doesn't work!
> > I would like META table can split by table name.
> > If I have 10 tables, there are 10 regions in META that can be
> > dispatched
> > to different region server, so I can start many
> > clients concurrently read different tables' data without META's
> > bottleneck.
> > I had started 2000 concurrent hbase java cleints to get only one
> > table's(187 regions ) META,
> > which took 302 sec.
> > What I want to do is to improve the META performance by splitting the
> > META.
> >
> >
> >
> >
> >
> > Fleming Chiu(邱宏明)
> > 707-6128
> > y_823910@tsmc.com
> > 週一無肉日吃素救地球(Meat Free Monday Taiwan)
> >
> >
> >
> >
> >
> >                       saint.ack@gmail.c
> >                       om                       To:      hbase-
> > user@hadoop.apache.org
> >                       Sent by:                 cc:      (bcc:
> > Y_823910/TSMC)
> >                       saint.ack@gmail.c        Subject: Re: Split
> META
> > manually
> >                       om
> >
> >
> >                       2010/03/12 02:01
> >                       PM
> >                       Please respond to
> >                       hbase-user
> >
> >
> >
> >
> >
> >
> > Why split .META.?  I'm not sure it works properly so would advise
> > against it (We don't have tests in place for that... we've not been
> > too concerned about it up to this since your install would have to be
> > massive for .META. to split).
> > St.Ack
> >
> > 2010/3/11  <y_823910@tsmc.com>:
> > > Hi there,
> > >
> > > I want to split META table manually but I wonder how to set the
> > optional
> > > Region Key in the webpage.
> > > (using the value like BIG_TABLE,FRPFXRD_NF61904-0
> > > 1.001.Main.0,1268309701214)
> > >
> > > BIG_TABLE,FRPFXRD_NF61904-0 column=info:regioninfo,
> > > timestamp=1268309711446, value=REGION => {NAME => 'BIG_TA
> > >  1.001.Main.0,1268309701214
> > > BLE,FRPFXRD_NF61904-01.001.Main.0,1268309701214', STARTKEY =>
> > > 'FRPFXRD_NF61904-01
> > >                             .001.Main.0', ENDKEY =>
> > > 'FRPFXRD_NG50972-01.001.Main.1', ENCODED => 1864321617, T
> > >
> > > There is only one region in our META table now , if I want to split
> > it by
> > > each table start key,
> > > is that possible? Thus it can release the META's bottleneck while
> > multiple
> > > clients reading request.
> > > Any ideas?
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Fleming Chiu(邱宏明)
> > > 707-6128
> > > y_823910@tsmc.com
> > > 週一無肉日吃素救地球(Meat Free Monday Taiwan)
> > >
> > >
> > >
> > ---------------------------------------------------------------------
> --
> > ----
> > >                                                         TSMC
> PROPERTY
> > >  This email communication (and any attachments) is proprietary
> > information
> > >  for the sole use of its
> > >  intended recipient. Any unauthorized review, use or distribution
> by
> > anyone
> > >  other than the intended
> > >  recipient is strictly prohibited.  If you are not the intended
> > recipient,
> > >  please notify the sender by
> > >  replying to this email, and then delete this email and any copies
> of
> > it
> > >  immediately. Thank you.
> > >
> > ---------------------------------------------------------------------
> --
> > ----
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >  --------------------------------------------------------------------
> --
> > -----
> >                                                          TSMC
> PROPERTY
> >  This email communication (and any attachments) is proprietary
> > information
> >  for the sole use of its
> >  intended recipient. Any unauthorized review, use or distribution by
> > anyone
> >  other than the intended
> >  recipient is strictly prohibited.  If you are not the intended
> > recipient,
> >  please notify the sender by
> >  replying to this email, and then delete this email and any copies of
> > it
> >  immediately. Thank you.
> >  --------------------------------------------------------------------
> --
> > -----
> >
> >
> 
> 
> 
> 
> 
> 
>  ----------------------------------------------------------------------
> -----
>                                                          TSMC PROPERTY
>  This email communication (and any attachments) is proprietary
> information
>  for the sole use of its
>  intended recipient. Any unauthorized review, use or distribution by
> anyone
>  other than the intended
>  recipient is strictly prohibited.  If you are not the intended
> recipient,
>  please notify the sender by
>  replying to this email, and then delete this email and any copies of
> it
>  immediately. Thank you.
>  ----------------------------------------------------------------------
> -----
> 
> 



Mime
View raw message