hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Create custom filter on HBase 0.96.1.1-cdh5.0.1
Date Thu, 21 Aug 2014 01:34:07 GMT
As you mentioned before, FilterProtos is used to perform parsing.

You can use the following command to find JIRAs which modified Filter.proto
:

git log hbase-protocol//src/main/protobuf/Filter.proto

One example is:

HBASE-7542 SCVF: Avoid sending two unwanted boolean values from client to RS

BTW please use pastebin - some spaces were missing in the code you pasted.

Cheers


On Wed, Aug 20, 2014 at 6:20 PM, gabriela montiel <
gabriela.montiel@oracle.com> wrote:

>
> Hi all,
>
> Here is the code snippet from the implementation of the parseFrom and
> toByteArray code. Is there something wrong I am doing there?
>
> Thanks,
>
>
> 1.
>    publicKeyValueFilter(byte[]key,byte[]value)
> 2.
>    {
> 3.
>    m_keyBytes=key;
> 4.
>    m_valueBytes=value;
> 5.
>    }
> 6.
> 7.
> 8.
>    /**
> 9.
>        * Filter the row if the K/V pair is not found in the row data.
> 10.
>        *
> 11.
>        * @param cell the cell in question
> 12.
>        *
> 13.
>        */
> 14.
>    publicFilter.ReturnCodefilterKeyValue(Cell cell)
> 15.
>    {
> 16.
>    if(Bytes.equals(m_keyBytes, cell.getQualifier())){//
>    CellUtil.cloneQualifier(cell))) {
> 17.
> 18.
>    byte[]valueBytes=cell.getValue();// CellUtil.cloneValue(cell);
> 19.
>    // found the matched k/v, do not filter this row
> 20.
>    if(Bytes.equals(m_valueBytes, valueBytes)){
> 21.
>    debug("filterKeyValue: row matches K/V pair");
> 22.
> 23.
>    m_bFilterRow=false;
> 24.
>    }
> 25.
>    }
> 26.
>    else{
> 27.
>    debug("filterKeyValue: row not matches key");
> 28.
>    }
> 29.
> 30.
>    // include all columns
> 31.
>    returnFilter.ReturnCode.INCLUDE;
> 32.
>    }
> 33.
> 34.
>    publicstatic Filter
>    parseFrom(byte[]pbBytes)throwsDeserializationException
> 35.
>    {
> 36.
>    try{
> 37.
>    ByteArrayInputStreambais=newByteArrayInputStream(pbBytes);
> 38.
>    DataInputStreaminput=newDataInputStream(bais);
> 39.
> 40.
>    debug("parseFrom: read key byte array from byte array");
> 41.
>    byte[]keyBytes=Bytes.readByteArray(input);
> 42.
> 43.
>    debug("parseFrom: read value byte array from byte array");
> 44.
>    byte[]valueBytes=Bytes.readByteArray(input);
> 45.
> 46.
>    returnnewKeyValueFilter(keyBytes, valueBytes);
> 47.
>    }
> 48.
>    catch(IOExceptione){
> 49.
>    thrownewDeserializationException(e);
> 50.
>    }
> 51.
>    }
> 52.
> 53.
>    publicbyte[]toByteArray()throwsIOException
> 54.
>    {
> 55.
>    ByteArrayOutputStreambaos=newByteArrayOutputStream(60/*size*/);
> 56.
>    DataOutputStreamdos=newDataOutputStream(baos);
> 57.
> 58.
>         debug("toByteArray: write key into byte array stream");
> 59.
>         Bytes.writeByteArray(dos, m_keyBytes);
> 60.
> 61.
>         debug("toByteArray: write value into byte array stream");
> 62.
>         Bytes.writeByteArray(dos, m_valueBytes);
> 63.
>         dos.flush();
> 64.
>         baos.flush();
> 65.
> 66.
>    returnbaos.toByteArray();
> 67.
>
>    }
>
>
>
>
> On 20/08/2014 06:46 p.m., Ted Yu wrote:
>
>> Can you show implementation for parseFrom(byte[]) - using pastebin ?
>>
>> If possible, seeing the code for whole class would help us understand
>> better.
>>
>> Cheers
>>
>>
>> On Wed, Aug 20, 2014 at 4:18 PM, gabriela montiel <
>> gabriela.montiel@oracle.com> wrote:
>>
>>  Hi all,
>>>
>>> I have been working on migrating a custom filter used in HBase 0.94 to
>>> make it work on HBase 0.96.1.1. This custom filter extends the FilterBase
>>> API and receives only two byte arrays. According to the documentation
>>> both
>>> toByteArray() and parseFrom(byte[]) should be implemented. After adding
>>> this APIs, copying the jar to all region servers and restarting HBase, I
>>> noticed that the filter is not working (all records are retrieved). I
>>> don't
>>> get any errors, but checking out the trace the constructor and
>>> toByteArray() APIs are only called but no other operation on the filter
>>> is
>>> executed.
>>>
>>> I saw that you need to use FilterProtos from the protobuf to write/read
>>> the byte array to rebuild the filter, however there is no documentation
>>> on
>>> how to update these APIs to add my custom filter. Do you have any idea on
>>> what we are doing wrong?
>>>
>>> Thanks,
>>>
>>> Gaby
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message