Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 20C4C18B21 for ; Thu, 3 Dec 2015 19:48:37 +0000 (UTC) Received: (qmail 49242 invoked by uid 500); 3 Dec 2015 19:48:36 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 49188 invoked by uid 500); 3 Dec 2015 19:48:36 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 49178 invoked by uid 99); 3 Dec 2015 19:48:36 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2015 19:48:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 37E811A0A7E for ; Thu, 3 Dec 2015 19:48:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.98 X-Spam-Level: *** X-Spam-Status: No, score=3.98 tagged_above=-999 required=6.31 tests=[FREEMAIL_REPLY=1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id bLpwg6kMe4pH for ; Thu, 3 Dec 2015 19:48:23 +0000 (UTC) Received: from BLU004-OMC4S16.hotmail.com (blu004-omc4s16.hotmail.com [65.55.111.155]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id AB60F21627 for ; Thu, 3 Dec 2015 19:48:22 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com ([65.55.111.137]) by BLU004-OMC4S16.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Thu, 3 Dec 2015 11:48:16 -0800 Received: from BN3PR13MB0817.namprd13.prod.outlook.com (10.165.78.17) by BN3PR13MB0819.namprd13.prod.outlook.com (10.165.78.19) with Microsoft SMTP Server (TLS) id 15.1.337.9; Thu, 3 Dec 2015 19:48:15 +0000 Received: from BN3PR13MB0817.namprd13.prod.outlook.com ([10.165.78.17]) by BN3PR13MB0817.namprd13.prod.outlook.com ([10.165.78.17]) with mapi id 15.01.0337.015; Thu, 3 Dec 2015 19:48:15 +0000 From: R P To: "user@flume.apache.org" Subject: Re: Context/Configuration values not passed to custom serializer. Thread-Topic: Context/Configuration values not passed to custom serializer. Thread-Index: AQHRLXOYkIBn9WNGc0aCHouPunTkuZ6472cAgAAVX1WAAAF5AIAApY6L Date: Thu, 3 Dec 2015 19:48:15 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: flume.apache.org; dkim=none (message not signed) header.d=none;flume.apache.org; dmarc=none action=none header.from=outlook.com; x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [WpXSK1N0Av8xPw0PAZmjmilKQgLzDsEI] x-microsoft-exchange-diagnostics: 1;BN3PR13MB0819;23:a8D7usRWBAjrpkd5RhPotP1zyOU5zSGMn9XaauUkpx6XoE7B+UiQhE0gAcSdZZoCMSMRriZME74yw5fnRIzvVQPgGOESa00GktHEMBJSKcE0KbEtUIRsYoepgfdDnZAxxbtSsxwF40J3ba51l/mjoFin6XiYDyJmgYRy7tD2k5FmN/Yh5Wi2hXe5xedlBPBkJ4Xb/nH+X+jCVvYzTY2wRQ==;5:pjOIHz36+/62hDtwn3iEYBkM4U0LIAsZuyyieB4TjegKRaKJbQl/rlCKBj9X57iA32y/tA4DxU086cD6MXnq/nTPR4kOPLLjvKm33foAhipOOxAmhw2KtjcBQjIzAa5Dp8+a3Pit+dzqKldZNindBQ==;24:WybrDO1EH19i7j/bVeMWbtmtI+CB9f2xgygh4Tze3PbJ3CaRyk4lSrWtunwPCsB/uA/JHYEyP17ieslI5n+Hf3uuU5sz07hngoS37eAPogI= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN3PR13MB0819; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(432015012)(82015046);SRVR:BN3PR13MB0819;BCL:0;PCL:0;RULEID:;SRVR:BN3PR13MB0819; x-forefront-prvs: 077929D941 x-forefront-antispam-report: SFV:NSPM;SFS:(7070004)(98900002);DIR:OUT;SFP:1901;SCL:1;SRVR:BN3PR13MB0819;H:BN3PR13MB0817.namprd13.prod.outlook.com;FPR:;SPF:None;LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_BN3PR13MB081718F856244495DDEC0054D60D0BN3PR13MB0817namp_" MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2015 19:48:15.0839 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR13MB0819 X-OriginalArrivalTime: 03 Dec 2015 19:48:16.0623 (UTC) FILETIME=[8D9ECFF0:01D12E03] --_000_BN3PR13MB081718F856244495DDEC0054D60D0BN3PR13MB0817namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thanks Gonzalo. I was missing serializer in my configuration, that's why it didn't appear i= n context. -R ________________________________ From: Gonzalo Herreros Sent: Thursday, December 3, 2015 1:54 AM To: user Subject: Re: Context/Configuration values not passed to custom serializer. Not sure how you are using the serializer but I understand it's part of an = hdfs sink When you do this: a1.sinks.k1.IgnoreNullKeyData=3DTrue your property goes to the sink. To make it apperar in the serializer context it should be like this: a1.sinks.k1.serializer.IgnoreNullKeyData=3DTrue How are you using the serializer in the configuration? Gonzalo On 3 December 2015 at 09:49, R P > wrote: Thanks for the reply Gonzalo. There is no configure method as Serializer doesn't extend Configurable. Do I have to extend Configurable to access custom configuration values? -R P.S - Sent code sample and config in separate email directly to you. ________________________________ From: Gonzalo Herreros > Sent: Thursday, December 3, 2015 12:32 AM To: user Subject: Re: Context/Configuration values not passed to custom serializer. It might be a bug in the sink you are using. For instance, I have a serializer for the HbaseSink so I added two custom p= roperties. tier1.sinks.hbase-sink-1.serializer.numberBuckets=3D20 tier1.sinks.hbase-sink-1.serializer.customKey=3Dtimestamp,type,resource,hos= tname Then in the configure method the context has the properties "numberBuckets"= and "customKey" (notice all the prefix of the property is removed and not = passed to the serializer) If you paste your configuration and code maybe we can spot where is your is= sue. Regards, Gonzalo On 3 December 2015 at 02:45, R P > wrote: Hello All, I need to use some configuration values in my implementation of Serializer= which extends only EventSerializer. For customizing behavior I am trying to use Context provided via Builder's = build method. Although the configuration values or none of the Context properties are not= passed to Serializer. I am able to use this Serializer without configuration through flume and wr= ite data so I am certain that my implementation works correctly. Any idea why Context is passed as empty to the Serializer? Or any pointers= on how I should go on debugging this issue? Thanks for reading and appreciate your response. Thanks, R --_000_BN3PR13MB081718F856244495DDEC0054D60D0BN3PR13MB0817namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable


Thanks Gonzalo. 
I was missing serializer in my configuration, that's why = it didn't appear in context. 

-R

From: Gonzalo Herreros <= gherreros@gmail.com>
Sent: Thursday, December 3, 2015 1:54 AM
To: user
Subject: Re: Context/Configuration values not passed to custom seria= lizer.
 

Not sure how you are using the serializer but I understand it's part of an = hdfs sink

When you do this:
a1.sinks.k1.IgnoreNullKeyData=3DTrue

your property goes to the sink.
To make it apperar in the serializer context it should be like this:

a1.sinks.k1.serializer.IgnoreNullKeyData=3DTrue

How are you using the serializer in the configur= ation?

Gonzalo


On 3 December 2015 at 09:49, R P <hadooper@o= utlook.com> wrote:

Thanks for the reply Gonzalo. 
There is no configure method as Serializer doesn't extend Configurable.&nbs= p;
Do I have to extend Configurable to access custom configuration values?

-R
P.S - Sent code sample and config in separate email directly to you.

From: Gonzalo Herreros <gherreros@gmail.com>
Sent: Thursday, December 3, 2015 12:32 AM
To: user
Subject: Re: Context/Configuration values not passed to custom seria= lizer.
 
It might be a bug in the sink you are using.
For instance, I have a serializer for the HbaseSink so I added two cus= tom properties.

tier1.sinks.hbase-sink-1.serializer.numberBuckets=3D20
tier1.sinks.hbase-sink-1.serializer.customKey=3Dtimestamp,type,resourc= e,hostname


Then in the configure method the context has the properties "numb= erBuckets" and "customKey" (notice all the prefix of the pro= perty is removed and not passed to the serializer)
If you paste your configuration and code maybe we can spot where is yo= ur issue.

Regards,
Gonzalo


On 3 December 2015 at 02:45, R P <hadooper@o= utlook.com> wrote:

Hello All,

 I need to use some configuration values in my implementation of Se= rializer which extends only EventSerializer. 

For customizing behavior I am trying to use Context provided via Bu= ilder's build method. 


Although the configuration values or none of the Context properties are = not passed to Serializer. 


I am able to use this Serializer without configuration through flume and write data so I am certain that my= implementation works correctly. 


Any idea why Context is passed as empty to the Thanks for reading and appreciate your response.  =


Thanks,

R





--_000_BN3PR13MB081718F856244495DDEC0054D60D0BN3PR13MB0817namp_--