nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Bende (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-936) Provenance Repository can become corrupt if it runs out of disk space, causing NiFi not to restart
Date Fri, 11 Sep 2015 17:30:46 GMT

    [ https://issues.apache.org/jira/browse/NIFI-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741195#comment-14741195
] 

Bryan Bende commented on NIFI-936:
----------------------------------

I started from scratch and rebuilt master with your patch and I have been getting fairly consistent
results now...

In the cases where I restart and keep the prov repo full, it either fails to start at all
because there is not any disk space, or if there is just a few kb it starts and quickly goes
back into the state it was in before where the partition is full.

In the cases where I free up some space and then start, it always starts up and the flow starts
working. One time during this scenario it generated the following exception while starting:

{code}
2015-09-11 13:14:50,363 ERROR [main] o.a.n.p.PersistentProvenanceRepository Failed to read
Provenance Event File /small/provenance_repository/279714.prov.gz due to {}
java.io.EOFException: Unexpected end of ZLIB input stream
    at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240) ~[na:1.7.0_65]
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158) ~[na:1.7.0_65]
    at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116) ~[na:1.7.0_65]
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) ~[na:1.7.0_65]
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254) ~[na:1.7.0_65]
    at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:41)
~[nifi-utils-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337) ~[na:1.7.0_65]
    at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.7.0_65]
    at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.7.0_65]
    at org.apache.nifi.provenance.StandardRecordReader.<init>(StandardRecordReader.java:91)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at org.apache.nifi.provenance.serialization.RecordReaders.newRecordReader(RecordReaders.java:106)
~[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:548)
[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:219)
[nifi-persistent-provenance-repository-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at org.apache.nifi.controller.FlowController.<init>(FlowController.java:406) [nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:349)
[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
[nifi-framework-core-0.3.0-SNAPSHOT.jar:0.3.0-SNAPSHOT]
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]

2015-09-11 13:14:57,168 INFO [main] o.a.n.p.PersistentProvenanceRepository Successfully merged
16 journal files (95531 records) into single Provenance Log File /small/provenance_repository/279714.prov
in 6785 milliseconds
2015-09-11 13:14:58,096 INFO [main] o.a.n.p.PersistentProvenanceRepository Successfully merged
16 journal files (134 records) into single Provenance Log File /small/provenance_repository/375261.prov
in 204 milliseconds
2015-09-11 13:15:05,590 INFO [main] o.a.n.p.PersistentProvenanceRepository Successfully merged
16 journal files (279714 records) into single Provenance Log File /small/provenance_repository/0.prov
in 7390 milliseconds
2015-09-11 13:15:05,785 INFO [main] o.a.n.p.PersistentProvenanceRepository Successfully merged
16 journal files (1084 records) into single Provenance Log File /small/provenance_repository/375411.prov
in 98 milliseconds
2015-09-11 13:15:05,835 INFO [main] o.a.n.p.PersistentProvenanceRepository Recovered 376495
records
2015-09-11 13:15:05,837 INFO [main] o.a.n.p.PersistentProvenanceRepository Created new Provenance
Event Writers for events starting with ID 376495
{code}

It continued to start up and worked fine, just making a note. 

If the above exception is not a concern, then I am a +1 as the code looks good, passes contrib
check, and I was unable to get into a bad state.

> Provenance Repository can become corrupt if it runs out of disk space, causing NiFi not
to restart
> --------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-936
>                 URL: https://issues.apache.org/jira/browse/NIFI-936
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 0.3.0
>
>         Attachments: 0001-NIFI-936-Ensure-that-if-we-fail-to-write-a-record-to.patch
>
>
> I caused the Provenance Repository to run out of disk space. When I attempted to restart,
I got a huge stack trace that ended with the following:
> {noformat}
> Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'flowController': FactoryBean threw exception on object creation; nested exception
is java.lang.OutOfMemoryError: Java heap space
> 	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
~[na:na]
> 	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
> 	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
~[na:na]
> 	... 123 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at org.apache.nifi.provenance.StandardRecordReader.readLongString(StandardRecordReader.java:427)
~[na:na]
> 	at org.apache.nifi.provenance.StandardRecordReader.readAttributes(StandardRecordReader.java:372)
~[na:na]
> 	at org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:314)
~[na:na]
> 	at org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1646)
~[na:na]
> 	at org.apache.nifi.provenance.PersistentProvenanceRepository.recoverJournalFiles(PersistentProvenanceRepository.java:1357)
~[na:na]
> 	at org.apache.nifi.provenance.PersistentProvenanceRepository.recover(PersistentProvenanceRepository.java:577)
~[na:na]
> 	at org.apache.nifi.provenance.PersistentProvenanceRepository.initialize(PersistentProvenanceRepository.java:219)
~[na:na]
> 	at org.apache.nifi.controller.FlowController.<init>(FlowController.java:406) ~[na:na]
> 	at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:349)
~[na:na]
> 	at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:63)
~[na:na]
> 	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[na:na]
> 	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1517)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:314)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:293)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
> 	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
~[na:na]
> 	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
~[na:na]
> 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
~[na:na]
> 	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
~[na:na]
> 	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222)
~[na:na]
> 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
~[na:na]
> 2015-09-08 07:48:28,165 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown
completed (nicely or otherwise).
> {noformat}
> Looking it over, it appears to be the same issue that caused problems with the FlowFile
repository in NIFI-902.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message