flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Context/Configuration values not passed to custom serializer.
Date Thu, 03 Dec 2015 09:54:18 GMT
Not sure how you are using the serializer but I understand it's part of an
hdfs sink

When you do this:
a1.sinks.k1.IgnoreNullKeyData=True

your property goes to the sink.
To make it apperar in the serializer context it should be like this:

a1.sinks.k1.serializer.IgnoreNullKeyData=True

How are you using the serializer in the configuration?

Gonzalo


On 3 December 2015 at 09:49, R P <hadooper@outlook.com> wrote:

> Thanks for the reply Gonzalo.
>
> There is no configure method as Serializer doesn't extend Configurable.
> Do I have to extend Configurable to access custom configuration values?
>
> -R
> P.S - Sent code sample and config in separate email directly to you.
>
> ------------------------------
> *From:* Gonzalo Herreros <gherreros@gmail.com>
> *Sent:* Thursday, December 3, 2015 12:32 AM
> *To:* user
> *Subject:* Re: Context/Configuration values not passed to custom
> serializer.
>
> It might be a bug in the sink you are using.
> For instance, I have a serializer for the HbaseSink so I added two custom
> properties.
>
> tier1.sinks.hbase-sink-1.serializer.numberBuckets=20
>
> tier1.sinks.hbase-sink-1.serializer.customKey=timestamp,type,resource,hostname
>
>
> Then in the configure method the context has the properties
> "numberBuckets" and "customKey" (notice all the prefix of the property is
> removed and not passed to the serializer)
> If you paste your configuration and code maybe we can spot where is your
> issue.
>
> Regards,
> Gonzalo
>
>
> On 3 December 2015 at 02:45, R P <hadooper@outlook.com> wrote:
>
>> Hello All,
>>
>>  I need to use some configuration values in my implementation of
>> Serializer which extends only EventSerializer.
>>
>> For customizing behavior I am trying to use Context provided via
>> Builder's build method.
>>
>>
>> Although the configuration values or none of the Context properties are
>> not passed to Serializer.
>>
>>
>> I am able to use this Serializer without configuration through flume and
>> write data so I am certain that my implementation works correctly.
>>
>>
>> Any idea why Context is passed as empty to the Serializer?  Or any
>> pointers on how I should go on debugging this issue?
>>
>>
>> Thanks for reading and appreciate your response.
>>
>>
>> Thanks,
>>
>> R
>>
>>
>>
>>
>

Mime
View raw message