flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From R P <hadoo...@outlook.com>
Subject Re: Context/Configuration values not passed to custom serializer.
Date Thu, 03 Dec 2015 19:48:15 GMT

Thanks Gonzalo.
I was missing serializer in my configuration, that's why it didn't appear in context.

-R

________________________________
From: Gonzalo Herreros <gherreros@gmail.com>
Sent: Thursday, December 3, 2015 1:54 AM
To: user
Subject: Re: Context/Configuration values not passed to custom serializer.


Not sure how you are using the serializer but I understand it's part of an hdfs sink

When you do this:
a1.sinks.k1.IgnoreNullKeyData=True

your property goes to the sink.
To make it apperar in the serializer context it should be like this:

a1.sinks.k1.serializer.IgnoreNullKeyData=True

How are you using the serializer in the configuration?

Gonzalo


On 3 December 2015 at 09:49, R P <hadooper@outlook.com<mailto:hadooper@outlook.com>>
wrote:

Thanks for the reply Gonzalo.

There is no configure method as Serializer doesn't extend Configurable.
Do I have to extend Configurable to access custom configuration values?

-R
P.S - Sent code sample and config in separate email directly to you.

________________________________
From: Gonzalo Herreros <gherreros@gmail.com<mailto:gherreros@gmail.com>>
Sent: Thursday, December 3, 2015 12:32 AM
To: user
Subject: Re: Context/Configuration values not passed to custom serializer.

It might be a bug in the sink you are using.
For instance, I have a serializer for the HbaseSink so I added two custom properties.

tier1.sinks.hbase-sink-1.serializer.numberBuckets=20
tier1.sinks.hbase-sink-1.serializer.customKey=timestamp,type,resource,hostname


Then in the configure method the context has the properties "numberBuckets" and "customKey"
(notice all the prefix of the property is removed and not passed to the serializer)
If you paste your configuration and code maybe we can spot where is your issue.

Regards,
Gonzalo


On 3 December 2015 at 02:45, R P <hadooper@outlook.com<mailto:hadooper@outlook.com>>
wrote:

Hello All,

 I need to use some configuration values in my implementation of Serializer which extends
only EventSerializer.

For customizing behavior I am trying to use Context provided via Builder's build method.


Although the configuration values or none of the Context properties are not passed to Serializer.


I am able to use this Serializer without configuration through flume and write data so I am
certain that my implementation works correctly.


Any idea why Context is passed as empty to the Serializer?  Or any pointers on how I should
go on debugging this issue?


Thanks for reading and appreciate your response.


Thanks,

R





Mime
View raw message