Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of
 ramkrishna.s.vasudevan@gmail.com designates 209.85.215.176 as permitted
 sender)
MIME-Version: 1.0
In-Reply-To: 
 <0CE69E9126D0344088798A3B7F7F80863AEB08D1@SZXEML553-MBX.china.huawei.com>
References: 
 <0CE69E9126D0344088798A3B7F7F80863AEA0620@SZXEML553-MBX.china.huawei.com>
	<CAF1+Vs8iVSBz9nDvhcPv74BDud3DPS9NC+Vm+Fs0DhDZW40X1A@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEA52D9@SZXEML553-MBX.china.huawei.com>
	<CANDVYV=5NH3VwdG4C4+z9qrQ1pet6bAgagNxz7DuQyY4b=NMpw@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEAABF3@SZXEML553-MBX.china.huawei.com>
	<CANDVYVnZQ6nUSj6GOpfitFK89To67xmQfRRLPWAjcAZv3A8ZpQ@mail.gmail.com>
	<CAOtJ30oYoNuWyeXnrTAoXS2YK0XSy3TZMXZbmm-NQ1qOEL9tiQ@mail.gmail.com>
	<CANDVYVneynqx4vvLWiRZGbZ0-pDy3VtGyDqdDsdCPtQtpQkFcw@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEAADDC@SZXEML553-MBX.china.huawei.com>
	<CAOT3TWpQZEFZSOXT-03Tq6X01tq17xo5JdfFqBLAdVnsi_kM5Q@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEABEA6@SZXEML553-MBX.china.huawei.com>
	<CANDVYVkEirR5qoxv+r4gh-W6NUoEekor9S5steduucUVraOftQ@mail.gmail.com>
	<CADt=Q7nUdOyT3sW4UbaxyBQK=BdYDL0uPKrrSqYAMSm7gh5iZg@mail.gmail.com>
	<CAOT3TWr5=wOWPmwLCo-nY51s6NnztgrK+gG9_RRZS_gwKAcsFA@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEB00CC@SZXEML553-MBX.china.huawei.com>
	<BLU0-SMTP3729947D6FCCFF588285328F240@phx.gbl>
	<1357691434.32597.YahooMailNeo@web140606.mail.bf1.yahoo.com>
	<CAF1+Vs9FzRdwxJHEAGss2wLhALdNP9-VnJg2+ei7Gv_tF1xeow@mail.gmail.com>
	<0CE69E9126D0344088798A3B7F7F80863AEB08D1@SZXEML553-MBX.china.huawei.com>
Date: Wed, 9 Jan 2013 09:41:46 +0530
Message-ID: 
 <CAAT7MkoVZfEF0dy0U5ZdRREypN-Z18hFtQJb+dyDW3gB=2MrAw@mail.gmail.com>
Subject: Re: HBase - Secondary Index
From: ramkrishna vasudevan <ramkrishna.s.vasudevan@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=047d7b34388eba814304d2d3429f

--047d7b34388eba814304d2d3429f
Content-Type: text/plain; charset=ISO-8859-1

As far as i can see its more related to using the coprocessor framework in
this soln that helps us in a great way to avoid unnecessary RPC calls when
we go with Region level indexing.

Regards
Ram

On Wed, Jan 9, 2013 at 8:52 AM, Anoop Sam John <anoopsj@huawei.com> wrote:

> Totally agree with Lars.  The design came up as per our usage and data
> distribution style etc.
> Also the put performance we were not able to compromise. That is why the
> region collocation based region based indexing design came :)
> Also as we are having the indexing and index usage every thing happening
> at server side, there is no need for any change in the client part
> depending on what type of client u use. Java code or REST APIs or any
> thing.  Also MR based parallel scans any thing can be comparably easy I
> feel as there is absolutely no changes needed at client side.  :)
>
> As Anil said there will be pros and cons for every way and which one suits
> your usage, needs to be adopted. :)
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Wednesday, January 09, 2013 6:58 AM
> To: user@hbase.apache.org; lars hofhansl
> Subject: Re: HBase - Secondary Index
>
> +1 on Lars comment.
>
> Either the client gets the rowkey from secondary table and then gets the
> real data from Primary Table. ** OR ** Send the request to all the RS(or
> region) hosting a region of primary table.
>
> Anoop is using the latter mechanism. Both the mechanism have their pros and
> cons. IMO, there is no outright winner.
>
> ~Anil Gupta
>
> On Tue, Jan 8, 2013 at 4:30 PM, lars hofhansl <larsh@apache.org> wrote:
>
> > Different use cases.
> >
> >
> > For global point queries you want exactly what you said below.
> > For range scans across many rows you want Anoop's design. As usually it
> > depends.
> >
> >
> > The tradeoff is bringing a lot of unnecessary data to the client vs
> having
> > to contact each region (or at least each region server).
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: Michael Segel <michael_segel@hotmail.com>
> > To: user@hbase.apache.org
> > Sent: Tuesday, January 8, 2013 6:33 AM
> > Subject: Re: HBase - Secondary Index
> >
> > So if you're using an inverted table / index why on earth are you doing
> it
> > at the region level?
> >
> > I've tried to explain this to others over 6 months ago and its not really
> > a good idea.
> >
> > You're over complicating this and you will end up creating performance
> > bottlenecks when your secondary index is completely orthogonal to your
> row
> > key.
> >
> > To give you an example...
> >
> > Suppose you're CCCIS and you have a large database of auto insurance
> > claims that you've acquired over the years from your Pathways product.
> >
> > Your primary key would be a combination of the Insurance Company's ID and
> > their internal claim ID for the individual claim.
> > Your row would be all of the data associated to that claim.
> >
> > So now lets say you want to find the average cost to repair a front end
> > collision of an S80 Volvo.
> > The make and model of the car would be orthogonal to the initial key.
> This
> > means that the result set containing insurance records for Front End
> > collisions of S80 Volvos would be most likely evenly distributed across
> the
> > cluster's regions.
> >
> > If you used a series of inverted tables, you would be able to use a
> series
> > of get()s to get the result set from each index and then find their
> > intersections. (Note that you could also put them in sort order so that
> the
> > intersections would be fairly straight forward to find.
> >
> > Doing this at the region level isn't so simple.
> >
> > So I have to again ask why go through and over complicate things?
> >
> > Just saying...
> >
> > On Jan 7, 2013, at 7:49 AM, Anoop Sam John <anoopsj@huawei.com> wrote:
> >
> > > Hi,
> > > It is inverted index based on column(s) value(s)
> > > It will be region wise indexing. Can work when some one knows the
> rowkey
> > range or NOT.
> > >
> > > -Anoop-
> > > ________________________________________
> > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > Sent: Monday, January 07, 2013 9:47 AM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > Hi Anoop,
> > >
> > > Am I correct in understanding that this indexing mechanism is only
> > > applicable when you know the row key? It's not an inverted index truly
> > > based on the column value.
> > >
> > > Mohit
> > > On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <anoopsj@huawei.com>
> > wrote:
> > >
> > >> Hi Adrien
> > >>                 We are making the consistency btw the main table and
> > >> index table and the roll back mentioned below etc using the CP hooks.
> > The
> > >> current hooks were not enough for those though..  I am in the process
> of
> > >> trying to contribute those new hooks, core changes etc now...  Once
> all
> > are
> > >> done I will be able to explain in details..
> > >>
> > >> -Anoop-
> > >> ________________________________________
> > >> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> > >> Sent: Monday, January 07, 2013 2:00 AM
> > >> To: user@hbase.apache.org
> > >> Subject: Re: HBase - Secondary Index
> > >>
> > >> Nice topic, perhaps one of the most important for 2013 :-)
> > >> I still don't get how you're ensuring consistency between index table
> > and
> > >> main table, without an external component (such as
> > bookkeeper/zookeeper).
> > >> What's the exact write path in your situation when inserting data ?
> > >> (WAL/RegionObserver, pre/post put/WALedit...)
> > >>
> > >> The underlying question is about how you're ensuring that WALEdit in
> > Index
> > >> and Main tables are perfectly sync'ed, and how you 're able to
> rollback
> > in
> > >> case of issue in both WAL ?
> > >>
> > >>
> > >> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <kelvin.msj@gmail.com>
> > >> wrote:
> > >>
> > >>>> Yes as you say when the no of rows to be returned is becoming more
> and
> > >>> more the latency will be becoming more.  seeks within an HFile block
> is
> > >>> some what expensive op now. (Not much but still)  The new encoding
> > >>> prefix
> > >>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > >> also
> > >>> presented this in the Hadoop China]  Thanks to Matt... :)  I am
> trying
> > to
> > >>> measure the scan performance with this new encoding . Trying to >back
> > >> port
> > >>> a simple patch for 94 version just for testing...   Yes when the no
> of
> > >>> results to be returned is more and more any index will become less
> > >>> performing as per my study  :)
> > >>>
> > >>> yes, you are right, I guess it's just a drawback of any index
> approach.
> > >>> Thanks for the explanation.
> > >>>
> > >>> Shengjie
> > >>>
> > >>> On 28 December 2012 04:14, Anoop Sam John <anoopsj@huawei.com>
> wrote:
> > >>>
> > >>>>> Do you have link to that presentation?
> > >>>>
> > >>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > >>>>
> > >>>> -Anoop-
> > >>>>
> > >>>> ________________________________________
> > >>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
> > >>>> Sent: Friday, December 28, 2012 9:12 AM
> > >>>> To: user@hbase.apache.org
> > >>>> Subject: Re: HBase - Secondary Index
> > >>>>
> > >>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <anoopsj@huawei.com
> >
> > >>>> wrote:
> > >>>>
> > >>>>> Yes as you say when the no of rows to be returned is becoming more
> > >> and
> > >>>>> more the latency will be becoming more.  seeks within an HFile
> block
> > >> is
> > >>>>> some what expensive op now. (Not much but still)  The new encoding
> > >>> prefix
> > >>>>> trie will be a huge bonus here. There the seeks will be flying..
> [Ted
> > >>>> also
> > >>>>> presented this in the Hadoop China]  Thanks to Matt... :)  I am
> > >> trying
> > >>> to
> > >>>>> measure the scan performance with this new encoding . Trying to
> back
> > >>>> port a
> > >>>>> simple patch for 94 version just for testing...   Yes when the no
> of
> > >>>>> results to be returned is more and more any index will become less
> > >>>>> performing as per my study  :)
> > >>>>>
> > >>>>> Do you have link to that presentation?
> > >>>>
> > >>>>
> > >>>>>> btw, quick question- in your presentation, the scale there is
> > >> seconds
> > >>> or
> > >>>>> mill-seconds:)
> > >>>>>
> > >>>>> It is seconds.  Dont consider the exact values. What is the % of
> > >>> increase
> > >>>>> in latency is important :) Those were not high end machines.
> > >>>>>
> > >>>>> -Anoop-
> > >>>>> ________________________________________
> > >>>>> From: Shengjie Min [kelvin.msj@gmail.com]
> > >>>>> Sent: Thursday, December 27, 2012 9:59 PM
> > >>>>> To: user@hbase.apache.org
> > >>>>> Subject: Re: HBase - Secondary Index
> > >>>>>
> > >>>>>> Didnt follow u completely here. There wont be any get()
> happening..
> > >>> As
> > >>>>> the
> > >>>>>> exact rowkey in a region we get from the index table, we can seek
> to
> > >>> the
> > >>>>>> exact position and return that row.
> > >>>>>
> > >>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> > >> just
> > >>>>> small number of rows returned, this works perfect. As you said you
> > >> will
> > >>>> get
> > >>>>> the exact rowkey positions per region, and simply seek them. I was
> > >>> trying
> > >>>>> to work out the case that when the number of result rows increases
> > >>>>> massively. Like in Anil's case, he wants to do a scan query against
> > >> the
> > >>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
> > >>> timestamp2"
> > >>>>> given no customerId provided. During that time period, he might
> have
> > >> a
> > >>>> big
> > >>>>> chunk of rows from different customerIds. The index table returns a
> > >> lot
> > >>>> of
> > >>>>> rowkey positions for different customerIds (I believe they are
> > >>> scattered
> > >>>> in
> > >>>>> different regions), then you end up seeking all different positions
> > >> in
> > >>>>> different regions and return all the rows needed. According to your
> > >>>>> presentation page14 - Performance Test Results (Scan), without
> index,
> > >>>> it's
> > >>>>> a linear increase as result rows # increases. on the other hand,
> with
> > >>>>> index, time spent climbs up way quicker than the case without
> index.
> > >>>>>
> > >>>>> btw, quick question- in your presentation, the scale there is
> seconds
> > >>> or
> > >>>>> mill-seconds:)
> > >>>>>
> > >>>>> - Shengjie
> > >>>>>
> > >>>>>
> > >>>>> On 27 December 2012 15:54, Anoop John <anoop.hbase@gmail.com>
> wrote:
> > >>>>>
> > >>>>>>> how the massive number of get() is going to
> > >>>>>> perform againt the main table
> > >>>>>>
> > >>>>>> Didnt follow u completely here. There wont be any get()
> happening..
> > >>> As
> > >>>>> the
> > >>>>>> exact rowkey in a region we get from the index table, we can seek
> > >> to
> > >>>> the
> > >>>>>> exact position and return that row.
> > >>>>>>
> > >>>>>> -Anoop-
> > >>>>>>
> > >>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> > >> kelvin.msj@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> how the massive number of get() is going to
> > >>>>>>> perform againt the main table
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> All the best,
> > >>>>> Shengjie Min
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> All the best,
> > >>> Shengjie Min
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Adrien Mogenet
> > >> 06.59.16.64.22
> > >> http://www.mogenet.me
> > >>
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>

--047d7b34388eba814304d2d3429f--