hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: parallel processing of regions in coprocessor
Date Thu, 02 Mar 2017 16:29:22 GMT
In production, I assume you have more than one server hosting the regions
you want to process.
Assuming load balancer does its job of spreading regions across servers,
you should consider using batchCoprocessorService.

There're only so many handlers on each region server, parallel invocation
of coprocessors of all relevant regions on individual server places burden
on the server.

FYI

On Thu, Mar 2, 2017 at 8:22 AM, Daniel Połaczański <dpolaczanski@gmail.com>
wrote:

> thx, this is what I needed
>
> 2017-03-02 11:07 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:
>
> > Daniel:
> > If you don't pass your ExecutorService to HTable ctor, the following
> would
> > be called:
> >
> >   public static ThreadPoolExecutor getDefaultExecutor(Configuration
> conf)
> > {
> >
> >     int maxThreads = conf.getInt("hbase.htable.threads.max", Integer.
> > MAX_VALUE);
> >
> > Note the default value.
> >
> > When the number of regions involved is high, consider adjusting the above
> > config.
> >
> > On Wed, Mar 1, 2017 at 11:04 PM, Daniel Połaczański <
> > dpolaczanski@gmail.com>
> > wrote:
> >
> > > Yes. I am using batch variant. I will try your variant
> > >
> > > 02.03.2017 7:53 AM "Jingcheng Du" <dujingch@gmail.com> napisał(a):
> > >
> > > What APIs are you using to execute the coprocessor?
> > > HTable#batchCoprocessorService? It executes the coprocessors regions
> by
> > > regions in each region server.
> > > How about using coprocessorService? This API should run coprocessors in
> > > parallel for different regions.
> > >
> > > 2017-03-02 14:32 GMT+08:00 Daniel Połaczański <dpolaczanski@gmail.com
> >:
> > >
> > > > I invoke my business logic which is similar to map reduce paradigm.
> > > > Everything works. Only performance is problem that regions are not
> > > > processed parallely.
> > > > I'm thinking about some hack like invoking 4 times rpc from client
> > when i
> > > > have 4 regions
> > > >
> > > > 02.03.2017 00:43 "Ted Yu" <yuzhihong@gmail.com> napisał(a):
> > > >
> > > > > To my knowledge, there is no support for this type of combination
> of
> > > map
> > > > > reduce and coprocessor.
> > > > >
> > > > > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański <
> > > > dpolaczanski@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > It is something like map reduce processing.
> > > > > > I want to run map and combine phase inside in a coprocessor
for
> > every
> > > > > > region separately, return partial results to the client. On
the
> > > client
> > > > > > invoke reduce phase and store result in another hbase table.
> > > > > >
> > > > > > 2017-03-01 23:26 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:
> > > > > >
> > > > > > > Can you describe your use case in more detail ?
> > > > > > >
> > > > > > > What type of custom coprocessor are you loading to the
region
> > > server
> > > > ?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > > > > > dpolaczanski@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > Let's assume that we have cluster consisting from
one
> > > RegionServer
> > > > > and
> > > > > > > the
> > > > > > > > RegionsServer contains one table consisting from 3
regions.
> > > > > > > >
> > > > > > > > I would like to process regions  in coprocessor parallely.
Is
> > it
> > > > > > > possible?
> > > > > > > >
> > > > > > > > I observed that currenlty it invokes coproprocessor
with all
> > the
> > > > > > regions
> > > > > > > > one by one. So it process at the beginig region1,
when
> finished
> > > > > process
> > > > > > > > regions2 and ...
> > > > > > > >
> > > > > > > > Is it possible to change this behaviour?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message