Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
Subject: Re: ConsumeKafka vs getKafka
MIME-Version: 1.0
From: Oleg Zhurakousky <ozhurakousky@hortonworks.com>
To: "dev@nifi.apache.org" <dev@nifi.apache.org>
Thread-Topic: ConsumeKafka vs getKafka
Thread-Index: AQHR4GscSgpxOWhocU+7aD7NYZ6U36AdGqz4
Date: Sun, 17 Jul 2016 20:56:57 +0000
Message-ID: <2788B4EF-CDE3-4AFB-A403-96CCFF5FD950@hortonworks.com>
References: <4996BE21-8664-4764-B32C-5736A9057DD2@gmail.com>
In-Reply-To: <4996BE21-8664-4764-B32C-5736A9057DD2@gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="windows-1251"
Content-Transfer-Encoding: quoted-printable
archived-at: Sun, 17 Jul 2016 20:57:16 -0000

Sumo

Thank you for pointing this out. Very valid comments and just in time for t=
he upcoming 1.0 release. Would you mind to raise a JIRA and we'll address i=
t promptly?=20

Cheers=20
Oleg

> On Jul 17, 2016, at 16:38, Sumanth Chinthagunta <xmlking@gmail.com> wrote=
:
>=20
> Env: Linux , Kafka 9.0.x, NiFi: 0.7=20
>=20
> We like to use new ConsumeKafka processor  but found critical limitations=
 compared to old getKafka processor.=20
> New ConsumeKafka is not writing critical Kafka attributes  i.e., kafka.ke=
y, kafka.offset, kafka.partition etc into flowFile attributes.=20
>=20
>=20
> Old getKafka processor:=20
>=20
> Standard FlowFile Attributes
> Key: 'entryDate'
>               Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'lineageStartDate'
>               Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'fileSize'
>               Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>               Value: '19709945781167274'
> Key: 'kafka.key'
>               Value: '{"database":"test","table":"sc_job","pk.systemid":1=
}'
> Key: 'kafka.offset'
>               Value: '1184010261'
> Key: 'kafka.partition'
>               Value: '0'
> Key: 'kafka.topic'
>               Value: =91data'
> Key: 'path'
>               Value: './'
> Key: 'uuid'
>               Value: '244059bb-9ad9-4d74-b1fb-312eee72124a'
>=20
> =97=97=97=97=97=97=97=97=97
> new ConsumeKafka processor :=20
>=20
> Standard FlowFile Attributes
> Key: 'entryDate'
>               Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'lineageStartDate'
>               Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'fileSize'
>               Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>               Value: '19710046870478139'
> Key: 'path'
>               Value: './'
> Key: 'uuid'
>               Value: '349fbeb3-e342-4533-be4c-424793fa5c59=92
>=20
> Thanks=20
> Sumo