gobblin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kidwell, Jack" <Jack_Kidwel...@comcast.com>
Subject RE: KafkaSimpleJsonExtractor
Date Thu, 12 Apr 2018 13:17:37 GMT
We want to extract both key and value from a Kafka message and publish the combined information
to HDFS. Keys contain numeric ids for the strings contained in values.

Please explain “other subscriber”. Are you proposing a different Kafka message structure?

KafkaSimpleJsonExtractor.java caught my attention because it extracts both keys and values
and puts them in decodedRecord.

From: Vicky Kak [mailto:vicky.kak@gmail.com]
Sent: Thursday, April 12, 2018 1:17 AM
To: user@gobblin.incubator.apache.org
Subject: Re: KafkaSimpleJsonExtractor

I am not sure what you are asking for. Do you want to see the keys/values rendered in the
logs while the extraction is being done?
Why can't you have the other subscriber to the kafka which will render the values rather than
KafkaExtractor implementation rendering the same?

On Wed, Apr 11, 2018 at 11:17 PM, Kidwell, Jack <Jack_Kidwelljr@comcast.com<mailto:Jack_Kidwelljr@comcast.com>>

Using gobblin_0.10.0, we want to use module, KafkaSimpleJsonExtractor.java in order to see
both kafka record keys and values.

How does one configure a job to achieve it?

View raw message