nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: How to recover from a full provenance repository?
Date Mon, 13 Mar 2017 22:23:05 GMT
Actually in following the referenced PR to the JIRA it references it
appears it went out in Apache NiFi 0.7.1 as
https://issues.apache.org/jira/browse/NIFI-2429.  So if you can get
onto that version it should address that condition.

Thanks
Joe

On Mon, Mar 13, 2017 at 3:20 PM, Joe Witt <joe.witt@gmail.com> wrote:
> James
>
> Yes the problem is likely to reappear most likely.  I suspect you're
> being impacted by https://issues.apache.org/jira/browse/NIFI-2395.
>
> It shows as though it was not addressed in any 0.x release, however,
> in looking at the JIRA notes it does appear that relevant aspects of
> this was merged into 0.x.  Joe Skora/Mike Moser: can you please review
> this JIRA and get it tagged to the proper fix version since it is now
> on master in 0.x?
>
> Thanks
> Joe
>
> On Mon, Mar 13, 2017 at 2:39 PM, James McMahon <jsmcmahon3@gmail.com> wrote:
>> Sent you that info Joe.
>>
>> Just to be certain I understood clearly: I should go ahead and rm -rf
>> ./provenance_repository? Won't the problem just reappear again as soon as it
>> recreates provenance_repository at start up?
>>
>> On Mon, Mar 13, 2017 at 5:04 PM, Joe Witt <joe.witt@gmail.com> wrote:
>>>
>>> Jim
>>>
>>> Blow away the prov repo and restart to get back going.  But to
>>> troubleshoot why it isn't cleaning up it seems as though there is some
>>> reason nifi is failing to rollover.
>>>
>>> What version precisely are you running?
>>>
>>> Thanks
>>> Joe
>>>
>>> On Mar 13, 2017 2:01 PM, "James McMahon" <jsmcmahon3@gmail.com> wrote:
>>>>
>>>> Here is all the info Joe. I had to manually recreate this here. Tried to
>>>> avoid typos. Anything that looks bizarre or oddly misspelled is on me.
>>>>
>>>> Certainly Joe. I need to do whatever is humanly possible to get this up
>>>> and
>>>> rolling again. Here is what I can provide:
>>>>
>>>> (Background)
>>>> /mnt/provenance_repo 50G size, 47G used, 17M available, 100% capacity
>>>> under this is
>>>> database_repository
>>>> provenance_repository
>>>> du -h output:
>>>> 8.5M ./database_repository
>>>> 16K ./lost+found
>>>> 4.0K ./provenance_repository/index-1489...124
>>>> 92K ./provenance_repository/toc
>>>> 47G ./provenance_repository/journals
>>>> 47G ./provenance_repository
>>>> 47G .
>>>>
>>>> In my nifi.properties:
>>>> #H2 Settings
>>>> nifi.database.directory=/mnt/provenance_repo/database_repository
>>>> nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
>>>>
>>>> nifi.provenance.repository.directory.default=/mnt/provenance_repo/provenance_repos
>>>> itory
>>>> nifi.provenance.repository.max.storage.time=24 hours
>>>> nifi.provenance.repository.max.storage.size=1 GB
>>>> nifi.provenance.repository.rollover.time=30 secs
>>>> nifi.provenance.repository.rollover.size=100 MB
>>>> nifi.provenance.repository.query.threads=2
>>>> nifi.provenance.repository.index.threads=1
>>>> nifi.provenance.repository.compress.on.rollover=true
>>>> nifi.provenance.repository.always.sync=false
>>>> nifi.provenance.repository.journal.count=16
>>>>
>>>> nifi.provenance.repository.indexed.fields=EventType,FlofFileUUID,Filename,ProcessorID,Relationship
>>>> nifi.provenance.repository.indexed.attributes=
>>>> nifi.provenance.repository.index.shard.size=500 MB
>>>> nifi.provenance.repository.max.attributes.length=65536
>>>> nifi.provenance.repository.buffer.size=100000
>>>> (Errors)
>>>> in nifi-bootstrap.log, java.io.IOException No space left on device
>>>> /mnt/provenance_repo/database_repository/nifi-users.h2.db
>>>> similar error reported in nifi-app.log, fails when trying to create
>>>> org.springframework.beans.factory.BeanCreationException
>>>>
>>>> I think both are just symptoms of the true problem, which is to say too
>>>> much stuff in
>>>> journals under provenance_repository
>>>> Jim
>>>>
>>>> On Mon, Mar 13, 2017 at 4:10 PM, Joe Witt <joe.witt@gmail.com> wrote:
>>>>>
>>>>> Jim,
>>>>>
>>>>> Processors create provenance events which get written to the
>>>>> provenenance repo but that is the end of their relationship.  The
>>>>> provenance repository is then self managing.  Can you share your
>>>>> nifi.properties provenance configuration?
>>>>>
>>>>> Thanks
>>>>> Joe
>>>>>
>>>>> On Mon, Mar 13, 2017 at 1:05 PM, James McMahon <jsmcmahon3@gmail.com>
>>>>> wrote:
>>>>> > Good evening. I have a number of process groups in my NiFi instance
>>>>> > that run
>>>>> > concurrently. Evidently I have filled up my provenance repo. How
do I
>>>>> > recover from this? NiFi will not start back up after shutdown.
>>>>> >
>>>>> > I am in a dev environment right now so I can afford to lose what
had
>>>>> > been in
>>>>> > my flows. Does that give me other options to recover?
>>>>> >
>>>>> > However I must determine what process group and processor are not
>>>>> > freeing
>>>>> > resources from the provenance repo. I can't allow this workflow
to
>>>>> > promote
>>>>> > to production until I determine how to prevent this from reoccurring.
>>>>> > How
>>>>> > can I do that?
>>>>> >
>>>>> > My provenance, content, and flow repos are all on separate disk
>>>>> > devices,
>>>>> > each of which is 50GB in capacity.
>>>>> >
>>>>> > Thank you for your help. -Jim
>>>>
>>>>
>>

Mime
View raw message