nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (Jira)" <>
Subject [jira] [Commented] (NIFI-6998) PutAzureEventHub to utilize Batch, Properties & Partitioning key
Date Tue, 14 Jan 2020 04:31:00 GMT


ASF subversion and git services commented on NIFI-6998:

Commit 2601f722b3230a640e86c83320a75b86288c0e60 in nifi's branch refs/heads/master from Martin
[;h=2601f72 ]

NIFI-6998 This closes #3972. Batch & Partitioning key
+ Asynchronous sending in limited batches
+ Property to determine attribute name carrying partitioning key
+ Maximum batch size property
+ Carrier object
- Unit test fakery
NIFI-6998 Attributes to User Defined properties
NIFI-6998 Unit tests
NIFI-6998 Review corrections
+ Interruption propagation (& test)
+ Final carrier members
+ Unnecessary generic modifiers removed from generic container
NIFI-6998 checkstyle corrections
+ Tabs to spaces, trailnig spaces
+ Absolute Imports
+ Braces locations
NIFI-6998 imports & license

Signed-off-by: Joe Witt <>

> PutAzureEventHub to utilize Batch, Properties & Partitioning key
> ----------------------------------------------------------------
>                 Key: NIFI-6998
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.8.0
>            Reporter: Martin Šimek
>            Priority: Major
>         Attachments: EH_01_flow.png, EH_02_flowFile.png, EH_03_processor_settings.png,
EH_04_EventHub.png, EH_05_receivedMessage.png
>   Original Estimate: 72h
>          Time Spent: 1h 40m
>  Remaining Estimate: 70h 20m
> Current implementation of _PutAzureEventHub_ is too simplistic. It sends one flow file
at a time synchronously which poses a significant bottlenec. It doesn't lift attributes into
the message's _User defined properties_ section. And most important problem is that you can't
specify the _partitioning key_ which plays critical role in a delivery consistency across
multiple partitions in case there are more partitions. Without partitioning key multiple use-cases
can't choose NiFi as an option.
> Aim of this improvement is to consume up to specified amount of flow files, lift all _attributes_
of each flow file into _User defined properties_ section of each message and let processor
to use one of those attributes as a partitioning key.
> Each flow file in a batch is sent asynchronously which is way more faster then processor
wait for all confirmations (or errors) and based on the result messages are transfered to
success or failure output.

This message was sent by Atlassian Jira

View raw message