nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon DeVries (JIRA)" <>
Subject [jira] [Updated] (NIFI-4775) Create a FlowFile repo backed by RocksDB
Date Thu, 15 Aug 2019 15:47:00 GMT


Brandon DeVries updated NIFI-4775:
    Attachment:     (was: rocksdb-flowfile-repo.adoc)

> Create a FlowFile repo backed by RocksDB
> ----------------------------------------
>                 Key: NIFI-4775
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Brandon DeVries
>            Priority: Major
>             Fix For: 1.10.0
>         Attachments: RocksDBFlowFileRepo.html
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
> Currently, when a FlowFile is written to the FlowFile Repository, the repo can either
fsync or not, depending on We should allow a third option, of fsync only
for CREATE events. In this case, if we receive new data from a source we can fsync the update
to the FlowFile Repository before ACK'ing the data from the source. This allows us to guarantee
data persistence without the overhead of an fsync for every FlowFile Repository update.
> It may make sense, though, to be a bit more selective about when do this. For example
if the source is a system that does not allow us to acknowledge the receipt of data, such
as a ListenUDP processor, this doesn't really buy us much. In such a case, we could be smart
about avoiding the high cost of an fsync. However, for something like GetSFTP where we have
to remove the file in order to 'acknowledge receipt' we can ensure that we wait for the fsync
before proceeding.
> NOTE: This functionality was ultimately provided in a new implementation backed by RocksDB

This message was sent by Atlassian JIRA

View raw message