nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sumanth Chinthagunta <xmlk...@gmail.com>
Subject Re: ConsumeKafka vs getKafka
Date Mon, 18 Jul 2016 03:44:56 GMT
Done.

https://issues.apache.org/jira/browse/NIFI-2298 <https://issues.apache.org/jira/browse/NIFI-2298>

Thanks 
-Sumo

> On Jul 17, 2016, at 1:37 PM, Sumanth Chinthagunta <xmlking@gmail.com> wrote:
> 
> Env: Linux , Kafka 9.0.x, NiFi: 0.7 
> 
> We like to use new ConsumeKafka processor  but found critical limitations compared to
old getKafka processor. 
> New ConsumeKafka is not writing critical Kafka attributes  i.e., kafka.key, kafka.offset,
kafka.partition etc into flowFile attributes. 
> 
> 
> Old getKafka processor: 
> 
> Standard FlowFile Attributes
> Key: 'entryDate'
>                Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'lineageStartDate'
>                Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'fileSize'
>                Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>                Value: '19709945781167274'
> Key: 'kafka.key'
>                Value: '{"database":"test","table":"sc_job","pk.systemid":1}'
> Key: 'kafka.offset'
>                Value: '1184010261'
> Key: 'kafka.partition'
>                Value: '0'
> Key: 'kafka.topic'
>                Value: ‘data'
> Key: 'path'
>                Value: './'
> Key: 'uuid'
>                Value: '244059bb-9ad9-4d74-b1fb-312eee72124a'
>  
> —————————
> new ConsumeKafka processor : 
>  
> Standard FlowFile Attributes
> Key: 'entryDate'
>                Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'lineageStartDate'
>                Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'fileSize'
>                Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>                Value: '19710046870478139'
> Key: 'path'
>                Value: './'
> Key: 'uuid'
>                Value: '349fbeb3-e342-4533-be4c-424793fa5c59’
> 
> Thanks 
> Sumo 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message