flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Hbase Sink
Date Tue, 10 Nov 2015 14:57:15 GMT
Sent without finishing the message:


I started by extending RegexHbaseEventSerializer, so I didn't have to
implement the full HbaseEventSerializer

Override (and call parent) configure(Context context) to get your custom
configuration properties

In
public void initialize(Event event, byte[] columnFamily)

you get the event and extract the data


In
public List<Row> getActions() throws FlumeException

you generate the HBase Put actions.


Regards,
Gonzalo

On 10 November 2015 at 14:55, Gonzalo Herreros <gherreros@gmail.com> wrote:

> I started by extending RegexHbaseEventSerializer, so I didn't have to
> implement the full HbaseEventSerializer
>
> Override (and call parent) configure(Context context) to get your custom
> configuration properties
>
> In
>
>
> public List<Row> getActions() throws FlumeException
>
> where you conver the
>
> On 7 November 2015 at 11:07, Rani Yaroshinski <rani.yaroshinski@gmail.com>
> wrote:
>
>> Any pointers to the code, as sample ?
>>
>> On Sat, Nov 7, 2015 at 12:45 PM, Gonzalo Herreros <gherreros@gmail.com>
>> wrote:
>>
>>> I did a custom serializer that parses the event an json and the top
>>> level properties become columns inside a configurable column family.
>>> I also have a custom property to configure which fields make up the
>>> composite key (which I salt based on the number of regions).
>>>
>>> It shouldn't be too hard having a property that maps input fields into
>>> hbase cf:column names (kinda like Hive does when you map an HBase table)
>>>
>>> Regards,
>>> Gonzalo
>>>
>>> On 7 November 2015 at 09:17, Rani Yaroshinski <
>>> rani.yaroshinski@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> It seems a little weird, but it looks like the most simple sink you
>>>> would like for hbase,
>>>> which allows you to set key, and columns generally is missing.
>>>> You need to write specific custom serializer for any implementation
>>>> which would make
>>>> a sense for any production system.
>>>> Anyone with pointers to rellevant custom implementations which allows
>>>> configuration for key
>>>> type and value, and also for column family data.
>>>>
>>>> Regards,
>>>> Rani
>>>>
>>>
>>>
>>
>

Mime
View raw message