nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: ConsumeKafka vs getKafka
Date Sun, 17 Jul 2016 20:56:57 GMT
Sumo

Thank you for pointing this out. Very valid comments and just in time for the upcoming 1.0
release. Would you mind to raise a JIRA and we'll address it promptly? 

Cheers 
Oleg

> On Jul 17, 2016, at 16:38, Sumanth Chinthagunta <xmlking@gmail.com> wrote:
> 
> Env: Linux , Kafka 9.0.x, NiFi: 0.7 
> 
> We like to use new ConsumeKafka processor  but found critical limitations compared to
old getKafka processor. 
> New ConsumeKafka is not writing critical Kafka attributes  i.e., kafka.key, kafka.offset,
kafka.partition etc into flowFile attributes. 
> 
> 
> Old getKafka processor: 
> 
> Standard FlowFile Attributes
> Key: 'entryDate'
>               Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'lineageStartDate'
>               Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'fileSize'
>               Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>               Value: '19709945781167274'
> Key: 'kafka.key'
>               Value: '{"database":"test","table":"sc_job","pk.systemid":1}'
> Key: 'kafka.offset'
>               Value: '1184010261'
> Key: 'kafka.partition'
>               Value: '0'
> Key: 'kafka.topic'
>               Value: ‘data'
> Key: 'path'
>               Value: './'
> Key: 'uuid'
>               Value: '244059bb-9ad9-4d74-b1fb-312eee72124a'
> 
> —————————
> new ConsumeKafka processor : 
> 
> Standard FlowFile Attributes
> Key: 'entryDate'
>               Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'lineageStartDate'
>               Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'fileSize'
>               Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>               Value: '19710046870478139'
> Key: 'path'
>               Value: './'
> Key: 'uuid'
>               Value: '349fbeb3-e342-4533-be4c-424793fa5c59’
> 
> Thanks 
> Sumo

Mime
View raw message