nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Payne (JIRA)" <>
Subject [jira] [Commented] (NIFI-220) Build Processors to interact with Kafka
Date Tue, 06 Jan 2015 00:56:35 GMT


Mark Payne commented on NIFI-220:


 - Delivery Guarantee does switch to a numeric value when running, as Kafka wants them as
-1, 0, or 1. But I didn't want to expose those arbitrary values in the UI, so I provided names.
 - Message Delimiter is a UTF-8 String. I considered requiring a hex-encoded byte string,
but that's pretty terrible if the data is text. Could make it configurable but then it's a
bit of a weird property to add. Could check if it starts with '0x' and if so expect a hex-encoded
byte sequence?

 - We do not store these values locally. If NiFi is restarted, it could result in duplication
of 'up to X amount of time worth of data'. Kafka's default is 60 seconds, so I chose to use
their default.
 - Message Demarcator: Same as Message Delimiter in PutKafka.

 - The Known Brokers vs. ZooKeeper Connection String, unfortunately, is what Kafka exposes.
Can't make them consistent with their API.
 - GetKafka is configured with a Demarcator, as it is demarcating the messages that it receives.
PutKafka has a delimiter, as the incoming data is delimited using this delimiter. I can see
your viewpoint, too, though... honestly, I think either way works.
 - Client Name is mapped to '' property in Kafka config. Their docs state "The client
id is a user-specified string sent in each request to help trace calls. It should logically
identify the application making the request." So modifying that should be fine.

I think the biggest thing that we want to consider here is the delimiter/demarcator: What
do you think of using a prefix of "0x" to indicate hex byte sequence, otherwise using a UTF-8
Encoded string?

> Build Processors to interact with Kafka
> ---------------------------------------
>                 Key: NIFI-220
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Task
>          Components: Extensions
>            Reporter: Mark Payne

This message was sent by Atlassian JIRA

View raw message