hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: A general question on maxVersion handling when we have Secondary index tables
Date Tue, 28 Aug 2012 17:34:02 GMT
I think we should revive secondary indexes discussion (actually it has been
revived)

Since Ramkrishna has design in mind, he would be the best person to log a
new JIRA.

Cheers

On Tue, Aug 28, 2012 at 10:03 AM, Jesse Yates <jesse.k.yates@gmail.com>wrote:

> @Ted: Are you proposing re-opening the should we have secondary indexes in
> HBase discussion? If so, I'm +1 on adding them. Wanna file a jira?
>
> @Wei Tan: Yeah, I generally agree. However, I think you can get away with
> ignoring MVCC and just keep an index on the latest key (where key
> _includes_ the timestamp) and then do lazy cleanup.
>
> @Ram: if you move the TS into the CQ you can remove the actual TS (so it
> costs you some minor computational overhead to pull it out), still giving
> you the right answer without actually using HBase timestamps.
>
> I've proposed that you can just do an async cleanup of the index when you
> find out its stale, with minimal overhead to the clients. Otherwise, yes,
> you would need a way to tie together the versions in the index and primary
> tables, which you don't always want to keep exactly the same.
>
> Also, there is an issue when returning the version of the row based on the
> indexed TS. Should you return the whole row? Should you return just the
> parts of the row with timestamps the same age or older? For the latter, how
> you do know which parts of the row to return when you have two versions of
> the same column that was indexed (which other row elements should be
> include based on TS)? I'd propose all questions that need to be answered if
> we are going to do a general hbase index.
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Tue, Aug 28, 2012 at 9:03 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > I think this discussion should be on HBASE JIRA.
> >
> > Another dimension to secondary indexing is the co-location (or pairing)
> of
> > data table region and index table region. Related regions from the two
> > tables should be placed on the same region server.
> >
> > Cheers
> >
> > On Tue, Aug 28, 2012 at 8:52 AM, Wei Tan <wtan@us.ibm.com> wrote:
> >
> > > Thanks for sharing a pointer to your implementation.
> > > My two cents:
> > > timestamp is a way to do MVCC and setting every KV with the same TS
> will
> > > get concurrency control very tricky and error prone, if not impossible
> > > I think Ram is talking about the dead entry in the index table rather
> > than
> > > data table. Deleting old index entries upfront when there is a new put
> > > might be a choice.
> > >
> > >
> > > Best Regards,
> > > Wei
> > >
> > > Wei Tan
> > > Research Staff Member
> > > IBM T. J. Watson Research Center
> > > 19 Skyline Dr, Hawthorne, NY  10532
> > > wtan@us.ibm.com; 914-784-6752
> > >
> > >
> > >
> > > From:   Jesse Yates <jesse.k.yates@gmail.com>
> > > To:     dev@hbase.apache.org,
> > > Date:   08/28/2012 04:00 AM
> > > Subject:        Re: A general question on maxVersion handling when we
> > have
> > > Secondary index tables
> > >
> > >
> > >
> > > Ram,
> > >
> > > If I understand correctly, I think you can design your index such that
> > you
> > > don't actually use the timestamp (e.g. everything gets put with a TS =
> 10
> > > -
> > > or some other non-special, relatively small number that's not 0 as I'd
> > > worry about that in HBase ;) Then when you set maxVersions to 1,
> > > everything
> > > should be good.
> > >
> > > You get a couple of wasted bytes from the TS, but with the prefixTrie
> > > stuff
> > > that should be pretty minimal overhead. If you do need to keep track of
> > > the
> > > timestamp you should be able to munge that back up into the column
> > > qualifier (and just know that that last 64 bits is the timestamp).
> Again
> > a
> > > little more CPU cost, but its really not that big of an overhead. It
> > seems
> > > like you don't really care about the TS though, in which case this
> should
> > > be pretty simple.
> > >
> > > Out of curiosity, what are people using for their secondary indexing
> > > solutions? I know there are a bunch out there, but don't know what
> people
> > > have adopted, what they like/dislike, design tradeoffs made and why.
> > >
> > > Disclaimer: I recently proposed a secondary indexing solution myself
> > > (shameless self-plug:
> > >
> > >
> >
> http://jyates.github.com/2012/07/09/consistent-enough-secondary-indexes.html
> > > )
> > > and its something I'm working on for Salesforce - open sourced at some
> > > point, promise!
> > >
> > > -Jesse
> > > -------------------
> > > Jesse Yates
> > > @jesse_yates
> > > jyates.github.com
> > >
> > >
> > > On Tue, Aug 28, 2012 at 12:24 AM, Ramkrishna.S.Vasudevan <
> > > ramkrishna.vasudevan@huawei.com> wrote:
> > >
> > > > Hi All
> > > >
> > > >
> > > >
> > > > When we try to build any type of secondary indices for a given table
> > how
> > > > can
> > > > one handle maxVersions in the secondary index tables.
> > > >
> > > >
> > > >
> > > > For eg,
> > > >
> > > > I have inserted
> > > >
> > > >  Row1  -  Val1  => t
> > > >
> > > > Row1 - Val2 => t+1
> > > >
> > > > Row1 - Val3. => t+2
> > > >
> > > >
> > > >
> > > > Ideally if my max versions is only one then Val3 should be my result
> If
> > > I
> > > > query on main table for row1.
> > > >
> > > >
> > > >
> > > > Now in my index I will be having all the above 3 entries.  Now how
> can
> > > we
> > > > remove the older entries from the index table that does not fit into
> > > > maxVersions.
> > > >
> > > >
> > > >
> > > > Currently while scanning and the code that avoids the max Versions
> does
> > > not
> > > > give any hooks to know the entries skipped thro versions.
> > > >
> > > > So any suggestions on this, I am still seeing the code for any other
> > > > options
> > > > but suggestions welcome.
> > > >
> > > >
> > > >
> > > > Regards
> > > >
> > > > Ram
> > > >
> > > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message