nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-220) Build Processors to interact with Kafka
Date Tue, 06 Jan 2015 00:56:35 GMT

    [ https://issues.apache.org/jira/browse/NIFI-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265447#comment-14265447
] 

Mark Payne commented on NIFI-220:
---------------------------------

Joe,

PutKafka:
 - Delivery Guarantee does switch to a numeric value when running, as Kafka wants them as
-1, 0, or 1. But I didn't want to expose those arbitrary values in the UI, so I provided names.
 - Message Delimiter is a UTF-8 String. I considered requiring a hex-encoded byte string,
but that's pretty terrible if the data is text. Could make it configurable but then it's a
bit of a weird property to add. Could check if it starts with '0x' and if so expect a hex-encoded
byte sequence?

GetKafka:
 - We do not store these values locally. If NiFi is restarted, it could result in duplication
of 'up to X amount of time worth of data'. Kafka's default is 60 seconds, so I chose to use
their default.
 - Message Demarcator: Same as Message Delimiter in PutKafka.

Combined:
 - The Known Brokers vs. ZooKeeper Connection String, unfortunately, is what Kafka exposes.
Can't make them consistent with their API.
 - GetKafka is configured with a Demarcator, as it is demarcating the messages that it receives.
PutKafka has a delimiter, as the incoming data is delimited using this delimiter. I can see
your viewpoint, too, though... honestly, I think either way works.
 - Client Name is mapped to 'client.id' property in Kafka config. Their docs state "The client
id is a user-specified string sent in each request to help trace calls. It should logically
identify the application making the request." So modifying that should be fine.


I think the biggest thing that we want to consider here is the delimiter/demarcator: What
do you think of using a prefix of "0x" to indicate hex byte sequence, otherwise using a UTF-8
Encoded string?

> Build Processors to interact with Kafka
> ---------------------------------------
>
>                 Key: NIFI-220
>                 URL: https://issues.apache.org/jira/browse/NIFI-220
>             Project: Apache NiFi
>          Issue Type: Task
>          Components: Extensions
>            Reporter: Mark Payne
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message