nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: setting processor concurrency based on the development/production environment
Date Thu, 01 Mar 2018 18:42:42 GMT
thanks Bryan and Kevin. I will be happy to open a jira - would it be a NiFi
jira or NiFi registry?

I like the approach that Bryan suggested.

I guess for now I will just color code the processors that need to be
changed in production.

P.S. I really, really like where NiFi is going...I've looked at StreamSets
and Cask, but for my purposes, I was looking for a tool when I can process
various tables without creating a flow per table. I was able to create a
very simple flow in NiFi, that will handle 25 tables. My next project is to
handle 600 tables in near real-time. I just could see how I would do that
with StreamSets or Cask, when you have to create a pipeline per table. I
was only being able to do something similar with Apache Airflow, but
airflow cannot do things in near real-time. The concept of FlowFiles with
attributes is a genius idea, and I am blown away with all the possibilities
to extend the functionality of NiFi with custom processors and Groovy
scripts. Awesome job, guys.

On Thu, Mar 1, 2018 at 1:29 PM, Kevin Doran <kdoran@apache.org> wrote:

> Hi Boris,
>
> Good point regarding concurrent tasks; thanks for sharing!
>
> This is a great candidate for something that one should be able to create
> environment-specific values for, as Bryan suggests. I agree we should
> create a NiFi JIRA to track this enhancement.
>
> Thanks,
> Kevin
>
> ´╗┐On 3/1/18, 11:44, "Bryan Bende" <bbende@gmail.com> wrote:
>
>     Hello,
>
>     Glad you are having success with NiFi + NiFi Registry!
>
>     You brought up an interesting point about the concurrent tasks...
>
>     I think we may want to consider making the concurrent tasks work
>     similar to variables, in that we capture the concurrent tasks that the
>     flow was developed with and would use it initially, but then if you
>     have modified this value in the target environment it would not
>     trigger a local change and would be retained across upgrades so that
>     you don't have to reset it.
>
>     For now you could probably always leave the versioned flow with the
>     lower value of 2, then once you are in prod you bump it to 4 until the
>     next upgrade is available, you then revert the local changes, do the
>     upgrade, and put it back to 4, but its not ideal because it shows a
>     local change the entire time.
>
>     I don't think there is much you can do differently right now, but I
>     think this is a valid case to create a JIRA for.
>
>     Thanks,
>
>     Bryan
>
>     On Thu, Mar 1, 2018 at 11:29 AM, Boris Tyukin <boris@boristyukin.com>
> wrote:
>     > Hello NiFi community,
>     >
>     > started using NiFi recently and fell in love with it! We run 1.6
> NiFi alone
>     > with new NiFi registry and I am trying to figure out how to promote
> NiFi
>     > flow, created in VM environment to our cluster.
>     >
>     > One of the things is "Concurrent Tasks" processor parameter. I bump
> it to 2
>     > or 4 for some processors in my flow, when I develop / test it in VM.
>     >
>     > Then we deploy this flow to a beefy cluster node (with 48 cores) and
> want to
>     > change concurrency to let's say 8 or 10 or 12 for some processors.
>     >
>     > Then I work on a new version/make some changes in my VM, and need to
> be more
>     > shy with concurrency so set it back to 2 or 4.
>     >
>     > Then the story repeats...
>     >
>     > Is there a better way than to manually set this parameter? I do not
> believe
>     > I can use a variable there and have to type the actual number of
> tasks.
>     >
>     >
>     > Thanks
>     > Boris
>
>
>
>

Mime
View raw message