hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sever Fundatureanu <fundatureanu.se...@gmail.com>
Subject Re: secondary index using coprocessors
Date Wed, 18 Jul 2012 15:09:34 GMT
Hi,

I also looked at your implementation. I have some issues with it but maybe
someone can correct me if I'm wrong.
What you're doing is writing the secondary index from the memstore in the
preFlush hook. However there is no mutual exclusive lock (write lock) that
is taken on the whole memstore for both the preFlush hook and the internal
flush operation. The only time a write lock is taken on the whole memstore
is when taking a snapshot and that comes after the preFlush hook.
So after completion of the preFlush hook, a couple more elements can be
added to the memstore until it is actually flushed to disk. This means you
lose consistency between your primary and your secondary index.
Can this issue be solved somehow? I am also interested in building a
secondary index from the preflush hook.

Thanks,
Sever

On Fri, Jul 6, 2012 at 6:04 AM, fding hbase <fding.hbase@gmail.com> wrote:

> Sorry, I don't have much experience on unit test. The performance may not
> be good
> since this is my leisure time work, nor do I have good understanding for
> consistency.
> I really wanna help but for now it's maybe beyond my ability. I'm still
> digging the code
> of hbase too... :-)
>
> HBase is quite HUGE to me...
>
> On Fri, Jul 6, 2012 at 10:32 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > The following action allows me to compile your code:
> >
> >   551  cp ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/*.java
> > src/main/java/org/apache/hadoop/hbase/regionserver/
> >   552  mkdir src/main/java/org/apache/hadoop/hbase/regionserver/indexed
> >   553  cp
> > ~/hbase-indexed/src/org/apache/hadoop/hbase/regionserver/indexed/*
> > src/main/java/org/apache/hadoop/hbase/regionserver/indexed/
> >
> > There is no unit test. Can you add some ?
> >
> > Do you have performance data: overhead for writing index, speedup in
> query
> > ?
> >
> > The most recent motion on this subject is HBASE-5723.
> > I still need to dig into your code. If you can comment on the level of
> > consistency guarantees for your design, that would be nice.
> >
> > Cheers
> >
> > On Thu, Jul 5, 2012 at 6:50 PM, fding hbase <fding.hbase@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > I've implemented an indexing system. It needs refinement of course
> > > but I have my own work to do, so I decided to open it for everyone to
> > > access, modify and any improvement or ideas would be great.
> > >
> > > https://github.com/danix800/hbase-indexed
> > >
> > > --
> > >
> > > Best Regards!
> > >
> > > Fei Ding
> > > fding.church@gmail.com
> > >
> >
>
>
>
> --
>
> Best Regards!
>
> Fei Ding
> fding.church@gmail.com
>



-- 
Sever Fundatureanu

Vrije Universiteit Amsterdam
E-mail: fundatureanu.sever@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message