nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-2732) ConsumeKafka 0.9 and 0.10 not handling partition reassignment case sufficiently
Date Tue, 06 Sep 2016 19:50:20 GMT

    [ https://issues.apache.org/jira/browse/NIFI-2732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15468354#comment-15468354
] 

ASF GitHub Bot commented on NIFI-2732:
--------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/nifi/pull/987


> ConsumeKafka 0.9 and 0.10 not handling partition reassignment case sufficiently
> -------------------------------------------------------------------------------
>
>                 Key: NIFI-2732
>                 URL: https://issues.apache.org/jira/browse/NIFI-2732
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Joseph Witt
>            Assignee: Joseph Witt
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> The new ConsumeKafka clients handle the threading model of the consumer api correctly.
 However, they are not yet honoring partition reassignment cases sufficiently which means
we could have avoidable cases of duplication.  By registering a partition reassignment listener
we can handle it correctly.
> Further, the processor is loading subsequent polls of messages into memory rather than
writing directly to the process session/disk.  We could write them to disk and achieve far
better performance and efficiency.  Early testing shows easily achieving 100MB/s sustained
per thread on a simple laptop setup with defaults which will scale very nicely on a legit
installed content repository.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message