nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-6998) PutAzureEventHub to utilize Batch, Properties & Partitioning key
Date Tue, 14 Jan 2020 04:31:00 GMT

    [ https://issues.apache.org/jira/browse/NIFI-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014852#comment-17014852
] 

ASF subversion and git services commented on NIFI-6998:
-------------------------------------------------------

Commit 2601f722b3230a640e86c83320a75b86288c0e60 in nifi's branch refs/heads/master from Martin
Šimek
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2601f72 ]

NIFI-6998 This closes #3972. Batch & Partitioning key
+ Asynchronous sending in limited batches
+ Property to determine attribute name carrying partitioning key
+ Maximum batch size property
+ Carrier object
- Unit test fakery
NIFI-6998 Attributes to User Defined properties
NIFI-6998 Unit tests
NIFI-6998 Review corrections
+ Interruption propagation (& test)
+ Final carrier members
+ Unnecessary generic modifiers removed from generic container
NIFI-6998 checkstyle corrections
+ Tabs to spaces, trailnig spaces
+ Absolute Imports
+ Braces locations
NIFI-6998 imports & license

Signed-off-by: Joe Witt <joewitt@apache.org>


> PutAzureEventHub to utilize Batch, Properties & Partitioning key
> ----------------------------------------------------------------
>
>                 Key: NIFI-6998
>                 URL: https://issues.apache.org/jira/browse/NIFI-6998
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.8.0
>            Reporter: Martin Šimek
>            Priority: Major
>         Attachments: EH_01_flow.png, EH_02_flowFile.png, EH_03_processor_settings.png,
EH_04_EventHub.png, EH_05_receivedMessage.png
>
>   Original Estimate: 72h
>          Time Spent: 1h 40m
>  Remaining Estimate: 70h 20m
>
> Current implementation of _PutAzureEventHub_ is too simplistic. It sends one flow file
at a time synchronously which poses a significant bottlenec. It doesn't lift attributes into
the message's _User defined properties_ section. And most important problem is that you can't
specify the _partitioning key_ which plays critical role in a delivery consistency across
multiple partitions in case there are more partitions. Without partitioning key multiple use-cases
can't choose NiFi as an option.
> Aim of this improvement is to consume up to specified amount of flow files, lift all _attributes_
of each flow file into _User defined properties_ section of each message and let processor
to use one of those attributes as a partitioning key.
> Each flow file in a batch is sent asynchronously which is way more faster then processor
wait for all confirmations (or errors) and based on the result messages are transfered to
success or failure output.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message