Mailing-List: contact users-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@nifi.apache.org
From: Stephen-Talk <stephen.schembri@talktalk.net>
Subject: Re: Is it possible to dynamically spawn a Processor Group ?
To: users@nifi.apache.org
References: <4b6252be-d427-c2a3-5f3b-2ded26566bd4@talktalk.net>
 <CAAvbYWXr+pkfNnK398Yjcd1Es8QmdALxq6tewxvyjJ48RBDpBA@mail.gmail.com>
 <785c3bb9-9a15-b22d-85d2-e795111d3b15@talktalk.net>
 <CAAvbYWUvvEqBixZpZjb2CiPUG=u7_oO6RLKLjjfDeBed7k7GRg@mail.gmail.com>
Message-ID: <66600a16-f3f2-f35c-69fd-8eb7c3b905de@talktalk.net>
Date: Sat, 8 Apr 2017 10:08:44 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <CAAvbYWUvvEqBixZpZjb2CiPUG=u7_oO6RLKLjjfDeBed7k7GRg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 08 Apr 2017 09:08:58 -0000

Jeff,

You hit the nail on the head.

My "Concurrent Tasks" is set to 1.

I shall have a fiddle with the numbers for both the threads, which is
set to 10 and the concurrent tasks, and see if it helps.

Thanks for your valuable assitance.

Have a great weekend...
Stephen

On 07/04/2017 18:58, Jeff wrote:
> Stephen,
> 
> It would be good to see screenshots of your flow, and get a little more
> information about your NiFi installation to help you get better
> throughput of your data.  Are you running NiFi as a single node?  At
> which processor in your flow are you noticing the queues backing up?  In
> the global settings menu, under "Controller Settings", what is "Maximum
> Timer Driven Thread Count" set to?  On your "MyImportProcessor" config,
> on the "Scheduling" tab, what is "Concurrent Tasks" set to?
> 
> In general terms, having more threads available to a processor means
> you'll get greater throughput of the data, provided that your IO
> configuration (disk read/write speed) can keep up.  The number of
> threads that NiFi is configured to use are made available to processors
> as flowfiles are presented to a processor via an incoming queue based on
> the number of concurrent tasks for which a processor is configured.
> 
> In the UI, you can see how many tasks are currently being executed by
> each processor, which will never be more than the "Maximum Timer Driven
> Thread Count" (for processors configured to use timer-based scheduling).
> 
> If you are experience backpressure on the incoming queue for
> "MyImportProcessor", try increasing the number of "Concurrent Tasks"
> available to that processor, and you may also want to increase the
> number of "Maximum Timer Driven Tread Count".
> 
> These are just some of the basics of getting more throughput in NiFi.
> 
> On Thu, Apr 6, 2017 at 4:25 PM Stephen-Talk
> <stephen.schembri@talktalk.net <mailto:stephen.schembri@talktalk.net>>
> wrote:
> 
>     Thanks for the quick reply.
> 
>     Yes, that is quite correct.
>     The scenario is the following:
> 
>     The input flow is a "GetFile" process that collects csv files
>     (>100,000 lines) which in turn queues the file and parses each line to a
>     locally built processor (MyImportProcessor say) that submits them via
>     the REST API to a Drupal website.
>     The process works fine, but it is very slow, and would like to speed it
>     up by splitting the csv file into chunks so that it can then spawn
>     "MyImportProcessor" as many times as required.
> 
> 
>     On 06/04/2017 20:47, Jeff wrote:
>     > Hello Stephen,
>     >
>     > It's possible to watch the status of NiFi, and upon observing a
>     > particular status in which you're interested, you can use the REST API
>     > to create new processor groups.  You'd also have to populate that
>     > processor group with processors and other components.  Based on the
>     > scenario you mentioned, though, it sounds like you are looking at
>     being
>     > able to scale up available processing (via more concurrent threads, or
>     > more nodes in a cluster) once a certain amount of data is queued
>     up and
>     > waiting to be processed, rather than adding components to the existing
>     > flow.  Is that correct?
>     >
>     > On Thu, Apr 6, 2017 at 3:30 PM Stephen-Talk
>     > <stephen.schembri@talktalk.net
>     <mailto:stephen.schembri@talktalk.net>
>     <mailto:stephen.schembri@talktalk.net
>     <mailto:stephen.schembri@talktalk.net>>>
>     > wrote:
>     >
>     >     Hi, I am just a Nifi Inquisitor,
>     >
>     >     Is it, or could it be possible to Dynamically spawn a
>     "Processor Group"
>     >     when the input flow reaches a certain threshold.
>     >
>     >     Thanking you in aniticipation.
>     >     Stephen
>     >
>