nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: Observation regarding PutKafka while implementing NIFI-1672
Date Wed, 13 Apr 2016 20:27:39 GMT
Pierre

You're absolutely correct. The explanation was valid, but there were problems with the original
implementation of how it was described, so that changed, but we didn't update the docs


> On Apr 13, 2016, at 16:24, Pierre Villard <pierre.villard.fr@gmail.com> wrote:
> 
> Hi,
> 
> I was working on NIFI-1672 [1], and while performing some tests with a
> local running Kafka instance, I noticed there is a possible error regarding
> how the behavior of the processor is documented.
> 
> The "Message delimiter" property is documented with:
> 
> "Specifies the delimiter (interpreted in its UTF-8 byte representation) to
> use for splitting apart multiple messages within a single FlowFile. If not
> specified, the entire content of the FlowFile will be used as a single
> message. If specified, the contents of the FlowFile will be split on this
> delimiter and each section sent as a separate Kafka message. Note that if
> messages are delimited and some messages for a given FlowFile are
> transferred successfully while others are not, the messages will be split
> into individual FlowFiles, such that those messages that were successfully
> sent are routed to the 'success' relationship while other messages are sent
> to the 'failure' relationship."
> 
> I believe that the part "Note that if messages are delimited and some
> messages for a given FlowFile are transferred successfully while others are
> not, the messages will be split into individual FlowFiles, such that those
> messages that were successfully sent are routed to the 'success'
> relationship while other messages are sent to the 'failure' relationship."
> is incorrect.
> 
> Instead I would say that the behavior (at least, it is what I observed) is:
> if one or multiple messages inside the FlowFile are not transferred
> successfully to Kafka, those messages are "tagged" thanks to some custom
> attributes of the FlowFile and the whole FlowFile is sent to 'failure'
> relationship. In case the 'failure' relationship is plugged back on the
> PutKafka processor, only the messages previously in error will be sent to
> Kafka.
> 
> I believe the documentation should be updated to reflect the current
> behavior. Should I do it as part of NIFI-1672? Or create a specific JIRA
> for that? In addition, should I create a JIRA to implement the behavior
> where FlowFile are splitted as currently documented?
> 
> Since Kafka processors have been source of discussions lately, advice and
> comments are welcomed!
> 
> Thanks !
> Pierre
> 
> [1] https://issues.apache.org/jira/browse/NIFI-1672

Mime
View raw message