nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noel Alex Makumuli <alexnoe...@gmail.com>
Subject Re: Application is no longer processing files
Date Wed, 26 Jul 2017 14:54:41 GMT
Hello Joe,
I think you have made a very good point here.. My directory had so many
files over 500k [lots of files], i was collecting them from different
sources.. then putting them in one folder
*Good point Pierre.  Could also be the api to list directories that have so
many files in them.  I recommend trying on a directory with less things to
list and see if that helps too as that means we need to evaluate other
things.*

But when i moved the director to the one which had around 100-150k then
everything is fine. I am just wondering why is it so because, i had set a
limit of 20 files for GetFile and then NIFI just could no longer respond
with a large file size.
I am not sure if i am able to replicate this problem or provide any clear
directions to show directions of a series of events which happened prior to
this problem.
All I remember is that I was changing the configuration to achieve maximum
speed of performing ETL, then the problem just showed up.

Thanks all for your input and quick responses to assist me on this matter.


On 26 July 2017 at 16:27, Joe Witt <joe.witt@gmail.com> wrote:

> Good point Pierre.  Could also be the api to list directories that
> have so many files in them.  I recommend trying on a directory with
> less things to list and see if that helps too as that means we need to
> evaluate other things.
>
> Also, we clearly could make it easier for users to see what is going
> on.  That it shows there is a thread on GetFile is cool but surfacing
> more details about what is going on would really help.  We need to
> have a reporting API for processors to build against so they can show
> more complex status data beyond what bulletins and such offer.
>
> On Wed, Jul 26, 2017 at 9:12 AM, Pierre Villard
> <pierre.villard.fr@gmail.com> wrote:
> > Hi Noel,
> >
> > Based on the configuration of the processor, NiFi is supposed to delete
> the
> > original file once retrieved by the processor (keep original file =
> false).
> > Is the user running NiFi allowed to delete the file? If not, as per the
> > documentation, the files will be ignored.
> >
> > Thanks
> > Pierre
> >
> >
> >
> > 2017-07-26 15:05 GMT+02:00 Juan Pablo Gardella
> > <gardellajuanpablo@gmail.com>:
> >>
> >> I had a similar issue but with ConsumerJMS.
> >>
> >> On Wed, 26 Jul 2017 at 10:04 Noel Alex Makumuli <alexnoel81@gmail.com>
> >> wrote:
> >>>
> >>> Hello Mark,
> >>> Thanks for the prompt response.
> >>> Yes the folder has the data.
> >>>
> >>>
> >>> /data/data_warehouse/data_sources/PCT/PCT_Backfile_non_
> Asian/nifi/sources$
> >>> find  -type f -iname wo\*.xml | wc -l
> >>> 157905
> >>>
> >>> The GetFile is not showing anything wrong run or attention that
> something
> >>> is missing.
> >>>
> >>> I duplicated the whole flow and changed the sources a but. It work only
> >>> worked for few thousands files and then it just no longer GetFile for
> >>> extraction.
> >>>
> >>> I am sending couple of images of the processor GetFile configuration
> and
> >>> you can advice further.
> >>>
> >>> Regards,
> >>> Noel
> >>>
> >>>
> >>>
> >>> On 26 July 2017 at 15:44, Mark Payne <markap14@hotmail.com> wrote:
> >>>>
> >>>> Noel,
> >>>>
> >>>> From the logs, it appears that you only have a handful of Processors.
> >>>> The only one that is a "source" processor
> >>>> is the GetFile. Are you sure that it is pointing to a directory that
> has
> >>>> data in it? Is it possible that you've got
> >>>> a typo there so that it isn't finding any data? Can you share the
> config
> >>>> for that processor?
> >>>>
> >>>> Thanks
> >>>> -Mark
> >>>>
> >>>>
> >>>> On Jul 26, 2017, at 7:37 AM, Noel Alex Makumuli <alexnoel81@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>> Hello,
> >>>> I am running Apache Nifi 1.3.0 on Ubuntu Server.
> >>>>
> >>>> The application is running fine and prints
> >>>> <code>
> >>>> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.BootstrapListener
> >>>> Successfully initiated communication with Bootstrap
> >>>> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.NiFi Controller
> >>>> initialization took 15369655939 nanoseconds.
> >>>> 2017-07-26 14:02:04,934 INFO [NiFi Web Server-50]
> >>>> c.s.j.s.i.application.WebApplicationImpl Initiating Jersey
> application,
> >>>> version 'Jersey: 1.19 02/11/2015 03:25 AM'
> >>>> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-5]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutFile[id=dee71437-10c9-115d-38ff-22e157972467] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,369 INFO [StandardProcessScheduler Thread-4]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutFile[id=dee7141f-10c9-115d-dcbf-430c652f8b48] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-3]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutFile[id=dee71430-10c9-115d-8a1a-d2e0355524d6] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-8]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutFile[id=dee71426-10c9-115d-701b-e65fcd56d485] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-5]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutFile[id=dee7141c-10c9-115d-ecc1-4fb0c3d2a5a3] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,374 INFO [StandardProcessScheduler Thread-4]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutFile[id=dee71433-10c9-115d-ff6c-a0160d72c5d4] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,373 INFO [StandardProcessScheduler Thread-6]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> EvaluateXPath[id=dee71422-10c9-115d-bb94-79ed7852b19b] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,375 INFO [StandardProcessScheduler Thread-8]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutFile[id=dee71428-10c9-115d-190e-c24016ed8015] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,376 INFO [StandardProcessScheduler Thread-3]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> ReplaceText[id=015d100b-704a-1013-58f4-955b7fe48f9c] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,381 INFO [StandardProcessScheduler Thread-7]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> GetFile[id=015d10c8-dee7-1411-5904-52201a329835] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,407 INFO [StandardProcessScheduler Thread-1]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> PutSQL[id=74971126-015d-1000-0838-8db7f5065da7] to run with 1 threads
> >>>> 2017-07-26 14:02:30,434 INFO [StandardProcessScheduler Thread-2]
> >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> >>>> TransformXml[id=791eee75-015d-1000-5c5d-df2bd5a4192b] to run with 1
> threads
> >>>> 2017-07-26 14:02:30,585 INFO [Flow Service Tasks Thread-1]
> >>>> o.a.nifi.controller.StandardFlowService Saved flow controller
> >>>> org.apache.nifi.controller.FlowController@10c992ce // Another save
> pending =
> >>>> false
> >>>> 2017-07-26 14:03:50,962 INFO [Write-Ahead Local State Provider
> >>>> Maintenance] org.wali.MinimalLockingWriteAheadLog
> >>>> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114
> Records
> >>>> and 0 Swap Files in 27 milliseconds (Stop-the-world time = 2
> milliseconds,
> >>>> Clear Edit Logs time = 1 millis), max Transaction ID
> >>>> 351
> >>>> 2017-07-26 14:03:52,616 INFO [pool-10-thread-1]
> >>>> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of
> FlowFile
> >>>> Repository
> >>>> 2017-07-26 14:03:53,064 INFO [pool-10-thread-1]
> >>>> org.wali.MinimalLockingWriteAheadLog
> >>>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705
> Records
> >>>> and 0 Swap Files in 448 milliseconds (Stop-the-world time = 132
> >>>> milliseconds, Clear Edit Logs time = 53 millis), max Transaction ID
> >>>> 1597176945
> >>>> 2017-07-26 14:03:53,065 INFO [pool-10-thread-1]
> >>>> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed
> FlowFile
> >>>> Repository with 6705 records in 448 milliseconds
> >>>> 2017-07-26 14:05:50,980 INFO [Write-Ahead Local State Provider
> >>>> Maintenance] org.wali.MinimalLockingWriteAheadLog
> >>>> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114
> Records
> >>>> and 0 Swap Files in 17 milliseconds (Stop-the-world time = 2
> milliseconds,
> >>>> Clear Edit Logs time = 2 millis), max Transaction ID
> >>>> 351
> >>>> 2017-07-26 14:05:53,065 INFO [pool-10-thread-1]
> >>>> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of
> FlowFile
> >>>> Repository
> >>>> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1]
> >>>> org.wali.MinimalLockingWriteAheadLog
> >>>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705
> Records
> >>>> and 0 Swap Files in 411 milliseconds (Stop-the-world time = 91
> milliseconds,
> >>>> Clear Edit Logs time = 63 millis), max Transaction ID 1597176945
> >>>> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1]
> >>>> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed
> FlowFile
> >>>> Repository with 6705 records in 411 milliseconds
> >>>> 2017-07-26 14:07:50,998 INFO [Write-Ahead Local State Provider
> >>>> Maintenance] org.wali.MinimalLockingWriteAheadLog
> >>>> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114
> Records
> >>>> and 0 Swap Files in 16 milliseconds (Stop-the-world time = 2
> milliseconds,
> >>>> Clear Edit Logs time = 1 millis), max Transaction ID
> >>>> 351
> >>>> 2017-07-26 14:07:53,478 INFO [pool-10-thread-1]
> >>>> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of
> FlowFile
> >>>> Repository
> >>>> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1]
> >>>> org.wali.MinimalLockingWriteAheadLog
> >>>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705
> Records
> >>>> and 0 Swap Files in 406 milliseconds (Stop-the-world time = 86
> milliseconds,
> >>>> Clear Edit Logs time = 55 millis), max Transaction ID 1597176945
> >>>> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1]
> >>>> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed
> FlowFile
> >>>> Repository with 6705 records in 406 milliseconds
> >>>> </code>
> >>>>
> >>>> The bootstrap is showing everything is normal only nifi-app.log is
> >>>> printing the above logs
> >>>>
> >>>> While the bootstrap.log is printing like below and nothing out of the
> >>>> ordinary or any exceptions.
> >>>> <code>
> >>>> ce/LOGS org.apache.nifi.NiFi
> >>>> 2017-07-26 14:01:34,005 INFO [main] org.apache.nifi.bootstrap.Command
> >>>> Launched Apache NiFi with Proc
> >>>> ess ID 12143
> >>>> 2017-07-26 14:01:34,705 INFO [NiFi Bootstrap Command Listener]
> >>>> org.apache.nifi.bootstrap.RunNiFi Apa
> >>>> che NiFi now running and listening for Bootstrap requests on port
> 32977
> >>>> </code>
> >>>>
> >>>> I changed couple of few things on the configuration mainly the memory
> to
> >>>> gain performance but nothing further.. I am no longer sure what is
> >>>> happening.
> >>>> onifi
> >>>> I have attached my log file , bootstrap & nifi configuration.
> >>>>
> >>>> on my configuration files..
> >>>> i disable the # The G1GC is still considered experimental but has
> proven
> >>>> to be very advantageous in providing great
> >>>> # performance without significant "stop-the-world" delays.
> >>>> #java.arg.13=-XX:+UseG1GC
> >>>>
> >>>> I saw in one thread people complaining of this.
> >>>>
> >>>> I have lots of files which need to get processed but nifi is just
> >>>> standing there while it should be ingesting the file.
> >>>> My last change on the nifi.properties file was::
> >>>> #  nifi.flow.configuration.archive.max.storage=500 MB
> >>>> nifi.flow.configuration.archive.max.storage=50 GB
> >>>> Could this be the culprit..?? Because reverting back does seem to help
> >>>> neither..!!
> >>>>
> >>>> I apologies if I have failed to frame my question correctly.
> >>>>
> >>>> Please advice.
> >>>> --
> >>>> NOEL ALEX MAKUMULI
> >>>>
> >>>> <bootstrap.txt><nifi.txt><nifi-app.log>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> NOEL ALEX MAKUMULI
> >>> TANZANIA
> >>>
> >>> Mobile: +255 755 100 700 ( Active)
> >>> Mobile: +358 44 3077 817 ( Active)
> >
> >
>



-- 
NOEL ALEX MAKUMULI
TANZANIA

Mobile: +255 755 100 700 ( Active)
Mobile: +358 44 3077 817 ( Active)

Mime
View raw message