nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: How does Nifi ingest large files?
Date Thu, 27 Oct 2016 15:58:03 GMT
That is correct.

Thanks
Joe

On Thu, Oct 27, 2016 at 11:55 AM, Jeremy Farbota <jfarbota@payoff.com>
wrote:

> Bryan,
>
> If I have the content repo implementation set to
> org.apache.nifi.controller.repository.VolatileContentRepository, it will
> stream the content in memory, correct?
>
> On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende <bbende@gmail.com> wrote:
>
>> Monica,
>>
>> Are you asking what does NiFi do when it picks up a large file from the
>> filesystem using a processor like GetFile?
>>
>> If so, it will stream the content of that file into NiFi's content
>> repository, and create a FlowFile pointing to that content. As far as NiFi
>> is concerned the content is just bytes at this point and has not been
>> changed in anyway from the original file.
>>
>> The content is not held in memory, and the FlowFile can move through many
>> processors without ever accessing the content, unless the processor needs
>> to, and then when accessing the content it is typically done in a streaming
>> fashion (when possible) to avoid loading the large content into memory.
>>
>> There are processors that can then split up the content based on specific
>> data formats, for example SplitText, SplitJSON, SplitAvro, etc.. but it is
>> up to the designer of the flow to do that.
>>
>> -Bryan
>>
>>
>> On Thu, Oct 27, 2016 at 4:52 AM, Monica Franceschini <
>> monica.franceschini@eng.it> wrote:
>>
>>> Hi,
>>> I'm figuring out how does Nifi ingest large files: does it split them
>>> into chunks or is it a massive load?Can you please, explain the behavior?
>>> Kind regards,
>>> Monica
>>> --
>>>
>>> *Monica Franceschini*
>>> Solution Architecture Manager
>>>
>>> *Big Data Competence Center Engineering Group*
>>> Corso Stati Uniti 23/C, 35127 Padova, Italia
>>> Tel: +39 049.8283547
>>> Fax: +39 049.8692566
>>> Twitter: @twittmonique
>>> www.spagobi.org - www.eng.it <http://www.eng.it/web/eng_en/home>     *proud
>>> SpagoBI supporter and contributor*
>>> [image: SpagoBI]
>>>   Respect the environment. Please don't print this e-mail unless you
>>> really need to.
>>>
>>> The information transmitted is intended only for the person or entity to
>>> which it is addressed and may contain confidential and/or privileged
>>> material. Any review, retransmission, dissemination or other use of, or
>>> taking of any action in reliance upon, this information by persons or
>>> entities other than the intended recipient is prohibited. If you received
>>> this in error, please contact the sender and delete the material from any
>>> computer.
>>>
>>
>>
>
>
> --
>
> [image: Payoff, Inc.] <http://www.payoff.com/>
>
> Jeremy Farbota
> Software Engineer, Data
> jfarbota@payoff.com <email@payoff.com> • (217) 898-8110 <(949)+430-0630>
>
> I'm a Storyteller. Discover your Financial Personality!
> <https://www.payoff.com/quiz>
>
> [image: Facebook]  <https://www.facebook.com/payoff> [image: Twitter]
> <https://www.twitter.com/payoff> [image: Linkedin]
> <https://www.linkedin.com/company/payoff-com>
>

Mime
View raw message