hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: Scan a region in parallel
Date Fri, 21 Oct 2016 08:55:49 GMT
Phoenix does support intelligent ways when you query using columns since it
is a SQL engine.

There the parallelism happens by using guideposts - those are fixed spaced
row keys stored in a seperate stats table. So when you do a query the
Phoenix internally spawns parallels scan queries using those guide posts
and thus making querying faster.

Regards
Ram

On Fri, Oct 21, 2016 at 1:26 PM, Anil <anilklce@gmail.com> wrote:

> Thank you Ram.
>
> "So now  you are spawning those many scan threads equal to the number of
> regions " - YES
>
> There are two ways of scanning region in parallel
>
> 1. scan a region with start row and stop row in parallel with single scan
> operation on server side and hbase take care of parallelism internally.
> 2. transform a start row and stop row of a region into number of start and
> stop rows (by some criteria) and span scan query for each start and stop
> row.
>
> #1 is not supported (as you also said).
>
> i am looking for #2. i checked the phoenix documentation and code. it seems
> to me that phoenix is doing #2. i looked into phoenix code and could not
> understand it completely.
>
> The usecase is very simple. Hbase not good (at least in terms of
> performance for OLTP) query by all columns (other than row key) and sorting
> of all columns of a row. even phoenix too.
>
> So i am planning load the hbase/phoenix table into in-memory data base for
> faster access.
>
> scanning of big region sequentially will lead to larger load time. so
> finding ways to minimize the load time.
>
> Hope this helps.
>
> Thanks.
>
>
> On 21 October 2016 at 09:30, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Hi Anil
> >
> > So now  you are spawning those many scan threads equal to the number of
> > regions.
> > bq.Is there any way to scan a region in parallel ?
> > You mean with in a region you want to scan parallely? Which means that a
> > single query you want to split up into N number of small scans and read
> and
> > aggregate on the client side/server side?
> >
> > Currently you cannot do that. Once you set a start and stoprow the scan
> > will determine which region it belongs to and retrieves the data
> > sequentially in that region (it applies the filtering that you do during
> > the course of the scan).
> >
> > Have you tried Apache Phoenix?  Its a SQL wrapper over HBase and there
> you
> > could do parallel scans for a given SQL query if there are some guide
> posts
> > collected. Such things cannot be an integral part of HBase. But I fear
> as I
> > am not aware of your usecase we cannot suggest on this.
> >
> > REgards
> > Ram
> >
> >
> > On Fri, Oct 21, 2016 at 8:40 AM, Anil <anilklce@gmail.com> wrote:
> >
> > > Any pointers ?
> > >
> > > On 20 October 2016 at 18:15, Anil <anilklce@gmail.com> wrote:
> > >
> > > > HI,
> > > >
> > > > I am loading hbase table into an in-memory db to support filter,
> > ordering
> > > > and pagination.
> > > >
> > > > I am scanning region and inserting data into in-memory db. each
> region
> > > > scan is done in single thread so each region is scanned in parallel.
> > > >
> > > > Is there any way to scan a region in parallel ? any pointers would be
> > > > helpful.
> > > >
> > > > Thanks
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message