nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Multiple Content Repositories
Date Thu, 26 Nov 2015 19:56:22 GMT
Chris,

It is something which occurs automatically and behind the scenes.
Under normal circumstances there will be many FlowFiles written to the
same content claim they'll just each have different offsets.  It is
more aligned with how disks work in terms of efficiently writing data,
efficiently reading data, and efficiently deleting the entire claim
(which is a file on disk).  Rather than a delete per flowfile we
delete once there are no more references to the entire claim.  Much
faster.  And all of that is totally abstracted away from the
perspective of someone writing extensions.  This bit, combined with
the copy on write and pass by reference logic the content repository
provides is a key part of what makes nifi efficient.

Thanks
Joe

On Thu, Nov 26, 2015 at 1:40 AM, Chris Lim <christopher.a.lim@gmail.com> wrote:
> Thanks Mark.
>
> The answer on the content repository round-robin is perfect. :)
>
> It got me curious when you mentioned that one or more FlowFiles can be
> written to the same Resource Claim. Is there a specific scenario wherein
> this can occur? Under normal circumstances there is only one FlowFile
> written to a Resource Claim?
>
> --
> Chris
>
>
> On Wed, Nov 25, 2015 at 9:39 PM, Mark Payne <markap14@hotmail.com> wrote:
>>
>> Chris,
>>
>> In terms of round robin-ing between the repositories, yes, it follows a
>> simple round-robin approach.
>> In terms of sections within those containers, the answer is more of a
>> "sort-of." Each FlowFile has what
>> we refer to as a Resource Claim, which points to a location in the content
>> repository. In the case of the
>> FileSystemRepository (which is the default and almost all that's ever used
>> right now), the Resource Claim
>> maps to a file on disk. In order to be very efficient, we may write many
>> FlowFiles to the same Resource Claim.
>>
>> Once we finish writing to a particular Resource Claim, we close the
>> resources and create a new one for the next
>> FlowFile. When we create these Resource Claims, we do so in a round-robin
>> fashion across the different Sections
>> of the content repository.
>>
>> Sorry, this is a fairly long-winded answer to such a seemingly simple
>> question :) but I wasn't sure how much detail you were
>> looking for. If anything is not clear, let us know.
>>
>> Thanks
>> -Mark
>>
>>
>> On Nov 25, 2015, at 5:12 AM, Chris Lim <christopher.a.lim@gmail.com>
>> wrote:
>>
>> Hi Guys,
>>
>> I am configuring our NiFi instance to have multiple content repositories
>> specifically with the "nifi.content.repository.directory." property setting
>> as mentioned in the Administrator's guide. Am I correct that flow file
>> contents are written to the repository using a round-robin algorithm? Also,
>> does the sections within a specific content repository follow the same
>> round-robin algorithm?
>>
>> Thanks,
>> Chris
>>
>>
>

Mime
View raw message