Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAJo9sUa_k0FvCFQR=Gnn=HSWJrFuHKnw-g5BgbjbSnymBPJHDg@mail.gmail.com>
References: 
 <CAGGEs2wjye6HHCqhcSaGV_97T+kBOYfN8yF1-JcR62VuDaEUPw@mail.gmail.com>
	<CALte62zCM7Xr8ge_p7+ceX0HGa=mt_KjC4jvfN_P_ddpO5zi-w@mail.gmail.com>
	<CAJuNEJAPw4LnTvSU2Z-TCB_AmQSHByhAnsxpf8GC05Kscyhq-w@mail.gmail.com>
	<CALte62wq=M8n+EoDc6cd-1PhO14tQHhwm5jacfSSq10rAJk3CA@mail.gmail.com>
	<CAJuNEJCFFTk2_YTXgwMW6Wzn81iH9KM=x_Aoiv+oFm7-oP00BA@mail.gmail.com>
	<CAGGEs2zauZNH0Bad9Yb7-wTf0X=cdUJcSYfvxjUhyYBofnC=UA@mail.gmail.com>
	<CAGGEs2wPgpiLStbGtvN4UWBezbfU5Ms=_=moyDb5V3r2jimq3g@mail.gmail.com>
	<CAJo9sUYGrOP0qSGnsNH6soTKTjkKgU8sp8aw02fnhQG7hRuwjQ@mail.gmail.com>
	<CAJo9sUb9B+4BHVJLGFF_kWmJX0cHhMjVFupTXVEt29bx+ivd_w@mail.gmail.com>
	<CAGGEs2yhTc8nXJ7TPpmTWp10+WTFbUmftBxKVHY4Fq14Ctcd_g@mail.gmail.com>
	<CAJo9sUb=b7s5WLwZJokhysa92BH=uK-0x2FCB87dH2OMyJUG9g@mail.gmail.com>
	<CAJo9sUa_k0FvCFQR=Gnn=HSWJrFuHKnw-g5BgbjbSnymBPJHDg@mail.gmail.com>
Date: Tue, 21 Oct 2014 18:53:24 -0700
Message-ID: 
 <CALte62ycg-R1iWjPLBAVX2nZ+FSyxwzkJrzfNJJqk6U+2nR4=g@mail.gmail.com>
Subject: Re: custom filter on hbase 0.96
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=089e013a15d09478d60505f93680

--089e013a15d09478d60505f93680
Content-Type: text/plain; charset=UTF-8

See this blog post:
http://www.flurry.com/2012/12/06/exploring-dynamic-loading-of-custom-filters-i#.VEcNtNR4rZg

Cheers

On Tue, Oct 21, 2014 at 6:48 PM, Kevin <kevin.macksamie@gmail.com> wrote:

> Also, if you do end up using dynamic loading, you'll need a way to version
> your filters because the RS will not reload a JAR if it changes.
>
> On Tue, Oct 21, 2014 at 9:46 PM, Kevin <kevin.macksamie@gmail.com> wrote:
>
> > I haven't tried dynamic loading of filters on RS, but I know it does
> > exist. See https://issues.apache.org/jira/browse/HBASE-9301.
> >
> > If you still can't get it to work, then I suggest distributing your
> > filters to the RS and restart them. Let us know how everything works out.
> >
> > On Tue, Oct 21, 2014 at 9:02 PM, Matt K <matvey1414@gmail.com> wrote:
> >
> >> Thanks Kevin!
> >>
> >> I was under impression, probably mistakingly, that as of 0.96 placing
> >> the filter on hdfs under hbase lib directory is sufficient and RS should
> >> load the filter dynamically from hdfs. Is that not the case?
> >>
> >> On Tuesday, October 21, 2014, Kevin <kevin.macksamie@gmail.com> wrote:
> >>
> >> > BTW, the error looks like you didn't distribute your custom filter to
> >> your
> >> > region servers.
> >> >
> >> > On Tue, Oct 21, 2014 at 1:34 PM, Kevin <kevin.macksamie@gmail.com
> >> > <javascript:;>> wrote:
> >> >
> >> > > Matt,
> >> > >
> >> > > You should create your own proto file and compile that with the
> Google
> >> > > Protocol Buffer compiler. Take a look at the
> SingleColumnValueFilter's
> >> > > code:
> >> > >
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
> >> > >
> >> > > You will need to override `public byte[] toByteArray()` and `public
> >> > static
> >> > > Filter parseFrom(byte[] pbBytes)`. The output of toByteArray()
> should
> >> be
> >> > > the byte[] from serializing with the protocol buffer. This will also
> >> be
> >> > the
> >> > > input to parseFrom(byte[]), which is called using reflection on the
> >> > > server-side to instantiate your custom filter and use it.
> >> > >
> >> > > On Sun, Oct 19, 2014 at 11:31 AM, Matt K <matvey1414@gmail.com
> >> > <javascript:;>> wrote:
> >> > >
> >> > >> Anyone?
> >> > >>
> >> > >> On Thursday, October 16, 2014, Matt K <matvey1414@gmail.com
> >> > <javascript:;>> wrote:
> >> > >>
> >> > >> > Hi, can anyone help with above? Feels like I'm missing something
> >> > >> obvious.
> >> > >> >
> >> > >> > On Wednesday, October 15, 2014, Nishanth S <
> >> nishanth.2884@gmail.com
> >> > <javascript:;>
> >> > >> > <javascript:_e(%7B%7D,'cvml','nishanth.2884@gmail.com
> >> <javascript:;>');>>
> >> > wrote:
> >> > >> >
> >> > >> >> Thanks Ted .I will take a look.
> >> > >> >>
> >> > >> >> -Nishanth
> >> > >> >>
> >> > >> >> On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu <yuzhihong@gmail.com
> >> > <javascript:;>> wrote:
> >> > >> >>
> >> > >> >> > Nishanth:
> >> > >> >> > Good question.
> >> > >> >> >
> >> > >> >> > As a general coding guide, writing unit test is always a good
> >> > start.
> >> > >> >> Using
> >> > >> >> > Matt's case as an example, take a look at TestPrefixFilter.
> >> > >> >> >
> >> > >> >> > There're various unit tests for Filters in hbase code.
> >> > >> >> >
> >> > >> >> > Cheers
> >> > >> >> >
> >> > >> >> > On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S <
> >> > nishanth.2884@gmail.com <javascript:;>
> >> > >> >
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > Hi Ted ,
> >> > >> >> > >  Since I am also working on similar thing is there a way we
> >> can
> >> > >> first
> >> > >> >> > test
> >> > >> >> > > the filter on client side?.You know what I  mean without
> >> > disrupting
> >> > >> >> > others
> >> > >> >> > > who are using the same cluster for other work?
> >> > >> >> > >
> >> > >> >> > > Thanks,
> >> > >> >> > > Nishanth
> >> > >> >> > >
> >> > >> >> > > On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu <
> yuzhihong@gmail.com
> >> > <javascript:;>>
> >> > >> wrote:
> >> > >> >> > >
> >> > >> >> > > > bq. Or create a new file, compile it into ...
> >> > >> >> > > >
> >> > >> >> > > > You should go with the above approach.
> >> > >> >> > > >
> >> > >> >> > > > On Wed, Oct 15, 2014 at 2:08 PM, Matt K <
> >> matvey1414@gmail.com
> >> > <javascript:;>>
> >> > >> >> wrote:
> >> > >> >> > > >
> >> > >> >> > > > > Hi all,
> >> > >> >> > > > >
> >> > >> >> > > > > I'm trying to get a custom filter to work on HBase 0.96.
> >> > After
> >> > >> >> some
> >> > >> >> > > > > searching, I found that starting from 0.96, the
> >> implementer
> >> > is
> >> > >> >> > required
> >> > >> >> > > > to
> >> > >> >> > > > > implement "toByteArray" and "parseFrom" methods, using
> >> > Protocol
> >> > >> >> > > Buffers.
> >> > >> >> > > > > But I'm having trouble with the "how".
> >> > >> >> > > > >
> >> > >> >> > > > > The "proto" file for the existing filters is located
> here:
> >> > >> >> > > > >
> >> > >> >> > > > >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >>
> >> >
> >>
> https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
> >> > >> >> > > > >
> >> > >> >> > > > > Am I supposed to modify that file? Or create a new file,
> >> > >> compile
> >> > >> >> it
> >> > >> >> > > into
> >> > >> >> > > > > Java, and package it up with the filter?
> >> > >> >> > > > >
> >> > >> >> > > > > In the meantime, I've taken a shortcut that's not
> working.
> >> > >> Here's
> >> > >> >> my
> >> > >> >> > > > code:
> >> > >> >> > > > > http://pastebin.com/iHFKu9Xz
> >> > >> >> > > > >
> >> > >> >> > > > > I'm using "PrefixFilter", which comes with HBase, since
> >> I'm
> >> > >> also
> >> > >> >> > > > filtering
> >> > >> >> > > > > by "prefix". However, that errors out with the
> following:
> >> > >> >> > > > > http://pastebin.com/zBg47p6Z
> >> > >> >> > > > >
> >> > >> >> > > > > Thanks in advance for helping!
> >> > >> >> > > > >
> >> > >> >> > > > > -Matt
> >> > >> >> > > > >
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >>
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> > www.calcmachine.com - easy online calculator.
> >> > >> >
> >> > >>
> >> > >>
> >> > >> --
> >> > >> www.calcmachine.com - easy online calculator.
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> www.calcmachine.com - easy online calculator.
> >>
> >
> >
>

--089e013a15d09478d60505f93680--