nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 尹文才 <batman...@gmail.com>
Subject Re: clear all flowfiles in all queues upon NiFi restart
Date Fri, 12 Jan 2018 04:15:51 GMT
Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, if I
switch to use VolatileFlowFileRepository, will NiFi swap out all the other
FlowFiles to disk if a queue is already full?
Is it just simply keeping all FlowFiles in memory?

Regards,
Ben

2018-01-12 12:07 GMT+08:00 尹文才 <batman713@gmail.com>:

> Thanks Mark, my case is that I'm using NiFi to do some ETL work and it's
> possible that NiFi dies unexpectedly due to lack of system resources. After
> NiFi restarts itself,
> I will re-extract all the data from database and re-perform all the
> operations, so I need to clear all possible FlowFiles that might exist in
> any queue.
>
> Regards,
> Ben
>
> 2018-01-12 11:49 GMT+08:00 Mark Payne <markap14@hotmail.com>:
>
>> Ben,
>>
>> I have to admit - that’s kind of an odd request :) I’m curious what the
>> use case is, if you can share?
>>
>> Regardless, the easiest way would be to update nifi.properties so that
>> the FlowFile repo that is used is the VolatileFlowFileRepository. This
>> would avoid writing the FlowFile state to disk, so ok restart you will lose
>> all FlowFiles. The content will still be present, but nifi will delete it
>> all on startup because there is no FlowFile associated with it.
>>
>> I’m on my phone right now so can’t easily tell you the exact name of the
>> property to change but you’ll probably find it pretty quickly. The Admin
>> Guide may well explain the different repositories as well.
>>
>> Thanks
>> -Mark
>>
>> Sent from my iPhone
>>
>> > On Jan 11, 2018, at 10:31 PM, 尹文才 <batman713@gmail.com> wrote:
>> >
>> > Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is
>> > restarted, but I don't know the correct way to do this. I checked all
>> > NiFi's guide documentation,
>> > it seems there're 2 possible solutions:
>> > 1. write a custom notification service: a notification service could be
>> > notified when NiFi is restarted and then inside the service, delete all
>> the
>> > files inside content_repository, flowfile_repository and
>> > provenance_repository.
>> >   I know there're now 2 existing services: email and http. But I'm not
>> > quite sure how to correctly write one and deploy it into my NiFi
>> > environment, is there a tutorial on writing one notification service?
>> >
>> > 2. I know from the developer guide that by using the annotation
>> @Shutdown
>> > in a custom processor, the method could be called when NiFi is
>> successfully
>> > shut down. The problem with this approach is the method could
>> >   not be guaranteed to be called when NiFi dies unexpectedly.
>> >
>> > Does anyone know what is the correct way to implement it? Thanks.
>> >
>> > Regards,
>> > Ben
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message