nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Weeks <swe...@weeksconsulting.us>
Subject Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Date Mon, 10 Feb 2020 18:41:55 GMT
The more I look at the error the more it looks like the container couldn't access the persistence
volume for some period of time. Can you share your yaml configuration minus sensitive stuff
and what environment your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.

Thanks
Shawn

´╗┐On 2/10/20, 12:27 PM, "Shawn Weeks" <sweeks@weeksconsulting.us> wrote:

    I've been testing out NiFi on K8s quite a bit lately and might be able to help. What are
you using for your persistent volume, this kinda sounds like NiFi can't always access data
in the content_repository folder?
    
    Thanks
    Shawn
    
    On 2/10/20, 11:53 AM, "Joe Witt" <joe.witt@gmail.com> wrote:
    
        ...sent a little too quick.  Also seems like if it is related to your k8s
        process that the issue would occur on older nifi releases too.  So that
        might be an interesting test for you.
        
        On Mon, Feb 10, 2020 at 9:52 AM Joe Witt <joe.witt@gmail.com> wrote:
        
        > Joe
        >
        > If you can replicate this in a non k8s environment that would likely make
        > resolution easier.  Far too many variables/questions at this stage.
        >
        > Thanks
        >
        > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <jgresock@gmail.com> wrote:
        >
        >> I don't know if there is something wrong with my Kubernetes-ized
        >> deployment
        >> of NiFi, but I'm seeing the following error crop up a lot in my NiFi
        >> 1.11.1
        >> deployment.  I can see the flow file attributes in "List Queue" for flow
        >> files affected by this problem, but the content claim is missing.
        >>
        >> It's not easily reproducible, but I've been running data constantly
        >> through
        >> the cluster for a few days, and every 2-4 hours or so, this problem
        >> appears
        >> in one of my processors (it's not always the same processors).  The error
        >> only goes away when I drain the relevant queue, but obviously that's not
        >> an
        >> option in production.
        >>
        >> Any thoughts on what might be causing intermittent truncations in the
        >> content_repository?
        >>
        >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
        >> o.a.nifi.processors.standard.ReplaceText
        >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
        >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
        >> iled to process session due to
        >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
        >> read
        >> from
        >>
        >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
        >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
        >> container=default, section=465], offset=0,
        >>
        >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
        >> Processor Administratively Yielded for 1 sec:
        >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
        >> read
        >> from
        >>
        >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
        >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
        >> container=default, section=465], offset=0,
        >>
        >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
        >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
        >> read
        >> from
        >>
        >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
        >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
        >> container=default, section=465], offset=0,
        >>
        >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
        >>         at
        >> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
        >>         at
        >>
        >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
        >>         at
        >>
        >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
        >>         at
        >>
        >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
        >>         at
        >>
        >> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
        >>         at
        >>
        >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
        >>         at
        >>
        >> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
        >>         at
        >>
        >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        >>         at
        >>
        >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
        >>         at
        >>
        >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
        >>         at
        >>
        >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
        >>         at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        >>         at
        >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        >>         at
        >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        >>         at
        >>
        >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        >>         at
        >>
        >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        >>         at
        >>
        >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        >>         at
        >>
        >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        >>         at java.lang.Thread.run(Thread.java:748)
        >> Caused by: java.io.EOFException: null
        >>         at org.apache.nifi.stream.io
        >> .StreamUtils.skip(StreamUtils.java:270)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .LimitedInputStream.read(LimitedInputStream.java:86)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
        >>         at
        >> org.apache.nifi.stream.io
        >> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
        >>         at java.io.FilterInputStream.read(FilterInputStream.java:133)
        >>         at
        >> org.apache.nifi.controller.repository.io
        >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
        >>         ... 20 common frames omitted
        >>
        >>
        >> Joe Gresock
        >>
        >
        
    
    

Mime
View raw message