Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <DM5PR14MB1273AA56CC348EA130655317AA6E0@DM5PR14MB1273.namprd14.prod.outlook.com>
References: <CY4PR14MB12721E51E01895443787F4CFAA6B0@CY4PR14MB1272.namprd14.prod.outlook.com>
 <CALte62y4kWYm3eYB01fiiMfhGtv81pFGWLUnu64DKvfa6gvwTw@mail.gmail.com>
 <DM5PR14MB1273DDCBDA5759B8AD118BBEAA6A0@DM5PR14MB1273.namprd14.prod.outlook.com>
 <CALte62w+AnniQzsaz5HbmfWTzRgOMwtpR7_U2A+_KHTgB5m1jQ@mail.gmail.com>
 <DM5PR14MB1273E8C0B7A5EB532D119D2CAA6F0@DM5PR14MB1273.namprd14.prod.outlook.com>
 <CALte62yvEfqGqOrmx8bwWAZmjrzeeR21euq4PMAu7M7DrAn2nw@mail.gmail.com>
 <DM5PR14MB127393D4B051A35F5CACB350AA6F0@DM5PR14MB1273.namprd14.prod.outlook.com>
 <84D6C4B7-6FD6-4106-BE19-C2EFAA0C97E7@gmail.com> <DM5PR14MB12738D11253B68EF3FD01482AA6E0@DM5PR14MB1273.namprd14.prod.outlook.com>
 <CALte62zNka9wv0zr34nNrBeP2Jm5=yCgdq9yBxxFaaRqusYVsw@mail.gmail.com> <DM5PR14MB1273AA56CC348EA130655317AA6E0@DM5PR14MB1273.namprd14.prod.outlook.com>
From: Ted Yu <yuzhihong@gmail.com>
Date: Mon, 9 Jan 2017 11:32:09 -0800
Message-ID: <CALte62zAiaeu3PucULNPt5h7qUjyuyXDS7dSOZC1d9aybgazVA@mail.gmail.com>
Subject: Re: Is it possible to implement a NOT filter in Hbase?
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c14665a76ed930545ae6c0b
archived-at: Mon, 09 Jan 2017 19:32:17 -0000

--94eb2c14665a76ed930545ae6c0b
Content-Type: text/plain; charset=UTF-8

Your first approach is more general (beneficial to more users).

However, looks like neither of us has the bandwidth to make it work.

On Mon, Jan 9, 2017 at 11:30 AM, Carl M <soloninguno@hotmail.com> wrote:

> Hi Ted,
>
>
> Just to tell that unfortunately i wasn't seeing any
> INCLUDE_AND_SEEK_NEXT_ROW nor INCLUDE_AND_NEXT_COL when debuging so adding
> them were not going to make any difference.
>
>
> Nevetherless i implement De Morgan's law as you second suggestion (it was
> easier as I expected) and it's working, so thanks again for that!
>
> Best,
>
> ________________________________
> De: Ted Yu <yuzhihong@gmail.com>
> Enviado: lunes, 02 de enero de 2017 11:36 p.m.
> Para: user@hbase.apache.org
> Asunto: Re: Is it possible to implement a NOT filter in Hbase?
>
> There is INCLUDE_AND_SEEK_NEXT_ROW which was not accounted for in the
> nested if statement.
>
> Is it possible for you to come up with unit test showing what you observe ?
>
> If you don't have time, please consider using De Morgan's law.
>
> Cheers
>
> On Mon, Jan 2, 2017 at 5:19 PM, Carl M <soloninguno@hotmail.com> wrote:
>
> > Thanks for your response Ted.
> >
> >
> > I did the change, unfortunately it doesn't make any difference.
> >
> >
> > Best,
> >
> > ________________________________
> > De: Ted Yu <yuzhihong@gmail.com>
> > Enviado: lunes, 02 de enero de 2017 07:58 p.m.
> > Para: user@hbase.apache.org
> > Asunto: Re: Is it possible to implement a NOT filter in Hbase?
> >
> > In the nested if statement, can you also handle NEXT_COL return code ?
> >
> > It should be translated to INCLUDE_AND_NEXT_COL
> >
> > Cheers
> >
> > > On Jan 2, 2017, at 1:54 PM, Carl M <soloninguno@hotmail.com> wrote:
> > >
> > > Sorry for not being clear enough.
> > >
> > >
> > > Maybe i misunderstand your suggestion, but what I've done in implement
> > my own filter, wrapping a FilterList
> > >
> > > public class NotFilterList extends Filter {
> > >    private FilterList filter;
> > >
> > >    public NotFilterList(final List<Filter> rowFilters) {
> > >        this.filter = new FilterList(rowFilters);
> > >    }
> > >
> > >    public void reset() throws IOException {
> > >        this.filter.reset();
> > >    }
> > >
> > >    public boolean filterRowKey(byte[] rowKey, int offset, int length)
> > throws IOException {
> > >        return this.filter.filterRowKey(rowKey, offset, length);
> > >    }
> > >
> > >    public boolean filterAllRemaining() throws IOException {
> > >        return this.filter.filterAllRemaining();
> > >    }
> > >
> > >    public ReturnCode filterKeyValue(Cell v) throws IOException {
> > >        ReturnCode code = this.filter.filterKeyValue(v);
> > >
> > >        if (code == ReturnCode.INCLUDE)
> > >            code = ReturnCode.SKIP;
> > >        else if (code == ReturnCode.INCLUDE_AND_NEXT_COL)
> > >            code = ReturnCode.NEXT_COL;
> > >        else if (code == ReturnCode.NEXT_ROW)
> > >            code = ReturnCode.INCLUDE_AND_NEXT_COL;
> > >        else if (code == ReturnCode.SKIP) {
> > >            code = ReturnCode.INCLUDE;
> > >        }
> > >
> > >        return code;
> > >    }
> > >
> > >    public Cell transformCell(Cell v) throws IOException {
> > >        return this.filter.transformCell(v);
> > >    }
> > >
> > >    public KeyValue transform(KeyValue currentKV) throws IOException {
> > >        return this.filter.transform(currentKV);
> > >    }
> > >
> > >    public void filterRowCells(List<Cell> kvs) throws IOException {
> > >        this.filter.filterRowCells(kvs);
> > >    }
> > >
> > >    public boolean hasFilterRow() {
> > >        return this.filter.hasFilterRow();
> > >    }
> > >
> > >    public boolean filterRow() throws IOException {
> > >        return this.filter.filterRow();
> > >    }
> > >
> > >    public KeyValue getNextKeyHint(KeyValue currentKV) throws
> IOException
> > {
> > >        return this.filter.getNextKeyHint(currentKV);
> > >    }
> > >
> > >    public Cell getNextCellHint(Cell currentKV) throws IOException {
> > >        return this.filter.getNextCellHint(currentKV);
> > >    }
> > >
> > >    public boolean isFamilyEssential(byte[] name) throws IOException {
> > >        return this.filter.isFamilyEssential(name);
> > >    }
> > >
> > >    public byte[] toByteArray() throws IOException {
> > >        return this.filter.toByteArray();
> > >    }
> > >
> > >    public static NotFilterList parseFrom(final byte [] pbBytes) throws
> > DeserializationException {
> > >        FilterList filterList = FilterList.parseFrom(pbBytes);
> > >        return new NotFilterList(filterList.getFilters());
> > >    }
> > >
> > >    boolean areSerializedFieldsEqual(Filter other) {
> > >        return this.filter.areSerializedFieldsEqual(other);
> > >    }
> > >
> > > }
> > >
> > >
> > > What I mean is that  filterKeyValue in the way i have it now, return
> the
> > right results but only the fields that were not originally skipped.
> > >
> > >
> > > So for example, if i have two rows each one with two fields
> > >
> > > Row 1
> > >
> > > Name: Bill
> > >
> > > Surname: Gates
> > >
> > >
> > > Row 2
> > >
> > > Name: Steve
> > >
> > > Surname: Jobs
> > >
> > >
> > > And I want to query for  Rows that doesn't have Name 'Bill'
> > >
> > > NOT (Name='Bill')
> > >
> > >
> > > What  I get as result from Hbase with this NotFilter is
> > >
> > > Row 2
> > >
> > > Surname: Jobs
> > >
> > >
> > > I suppose it's related to the cell "Name: Steve" skipped in the first
> > place (before reversing the ReturnCode).
> > >
> > >
> > > Best,
> > >
> > >
> > >
> > > ________________________________
> > > De: Ted Yu <yuzhihong@gmail.com>
> > > Enviado: lunes, 02 de enero de 2017 06:31 p.m.
> > > Para: user@hbase.apache.org
> > > Asunto: Re: Is it possible to implement a NOT filter in Hbase?
> > >
> > > bq. the cell/value that was originally skip is not return
> > >
> > > Can you be a bit more specific (with a concrete example) : the skipped
> > cell
> > > would not be returned (as indicated by the ReturnCode).
> > >
> > > Thanks
> > >
> > >> On Mon, Jan 2, 2017 at 1:06 PM, Carl M <soloninguno@hotmail.com>
> wrote:
> > >>
> > >> Hi Ted,
> > >>
> > >>
> > >> I tried your suggestion, unfortunately it doesn't work as expected. I
> > >> don't fully understand FilterList, but if a cell value was skip and I
> > >> reverse the ReturnCode, i get the right row but the cell/value that
> was
> > >> originally skip is not return.
> > >>
> > >> I also tried reversing only filterRow() method of FilterList, but I
> got
> > >> the same behaviour (the original cell/value missing).
> > >>
> > >>
> > >> Best,
> > >>
> > >>
> > >> ________________________________
> > >> De: Ted Yu <yuzhihong@gmail.com>
> > >> Enviado: viernes, 30 de diciembre de 2016 12:56 p.m.
> > >> Para: user@hbase.apache.org
> > >> Asunto: Re: Is it possible to implement a NOT filter in Hbase?
> > >>
> > >> I think the ReturnCode opposite INCLUDE_AND_NEXT_COL is NEXT_COL :
> > you're
> > >> not interested in any version of the current Cell.
> > >>
> > >> Cheers
> > >>
> > >>> On Fri, Dec 30, 2016 at 4:53 AM, Carl M <soloninguno@hotmail.com>
> > wrote:
> > >>>
> > >>> Thanks Ted! Great idea replacing the value in filterKeyValue.
> Although
> > >> I'm
> > >>> not quite sure looking at FilterList code if only INCLUDE/SKIP should
> > be
> > >>> replaced, and which should be the correct replacement for
> > >>> INCLUDE_AND_NEXT_COL. What do you think? If not maybe i should try to
> > >>> implement DeMorgan's law but I think it would be harder.
> > >>>
> > >>>
> > >>> Best,
> > >>>
> > >>> ________________________________
> > >>> De: Ted Yu <yuzhihong@gmail.com>
> > >>> Enviado: jueves, 29 de diciembre de 2016 06:10 p.m.
> > >>> Para: user@hbase.apache.org
> > >>> Asunto: Re: Is it possible to implement a NOT filter in Hbase?
> > >>>
> > >>> You can try negating the ReturnCode from filterKeyValue() (at the
> root
> > of
> > >>> FilterList):
> > >>>
> > >>>  abstract public ReturnCode filterKeyValue(final Cell v) throws
> > >>> IOException;
> > >>>
> > >>> INCLUDE -> SKIP
> > >>>
> > >>> SKIP -> INCLUDE
> > >>>
> > >>> Alternatively, you can use De Morgan's law to transfer the condition:
> > >>>
> > >>> NOT (a = '123' AND b = '456') becomes
> > >>>
> > >>> (NOT a = '123') OR (b = '456')
> > >>>
> > >>>> On Thu, Dec 29, 2016 at 12:56 PM, Carl M <soloninguno@hotmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi guys
> > >>>>
> > >>>>
> > >>>> I'm trying to implement some kind of NOT filter in Hbase, but don't
> > >> know
> > >>>> if possible, I'm playing with FilterIfMissing and
> FilterList.Operator
> > >> but
> > >>>> without luck.
> > >>>>
> > >>>>
> > >>>> I know how to return rows not having a specific column, but I mean
> > >>>> something like returning rows NOT fullfilling a condition, where
> > >>> condition
> > >>>> could be not only a SingleColumnValueFilter but a combined condition
> > >> with
> > >>>> FilterList. In SQL would be something like this for example
> > >>>>
> > >>>>
> > >>>> SELECT * FROM table WHERE NOT (a = '123' AND b = '456');
> > >>>>
> > >>>>
> > >>>> Thanks in advance,
> > >>
> >
>

--94eb2c14665a76ed930545ae6c0b--