nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-1822) Allow concurrent execution of ExecuteScript
Date Tue, 17 May 2016 21:59:13 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287701#comment-15287701
] 

ASF subversion and git services commented on NIFI-1822:
-------------------------------------------------------

Commit c13fb3159f59778e78769a1ad3a9f7131cfb7d96 in nifi's branch refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=c13fb31 ]

NIFI-1822 Enabled pooled ExecuteScript engines to facilitate concurrent processing.

Removed unused variable in unit test. (+10 squashed commits)
Squashed commits:
[7c5acc1] NIFI-1822 Removed trailing whitespace to conform with checkstyle rules.
[cb108cd] NIFI-1822 Added ASF License to unit test.
[9264428] NIFI-1822 Removed debugging log statements for script engine queue size.
Added unit tests demonstrating pooled execution timing and thread usage.
[bdbc4ba] NIFI-1822 Renamed reference to MockProcessorContext#setNumThreads to setMaxConcurrentTasks
after refactor.
[12bbe82] NIFI-1822 Moved unit test to correct directory.
Added test script resource which generates flowfile and updates attribute with current thread.
Added tests for single run, serial run, and pooled run (not complete).
[4c174c8] NIFI-1822 Added debugging messages to script execution.
[8ab0ce5] NIFI-1822 Added for loop to instantiate multiple script engines in queue.
[8c5ba51] NIFI-1822 Added variable max concurrent task field in MockProcessorContext because
it was previously hardcoded to 1. Changed setNumThreads to setMaxConcurrentTasks to maintain
naming convention.
[fd9120c] NIFI-1822 Added unit test skeleton for pooled script processor execution.
[23e4f68] NIFI-1822: Allow concurrent execution in ExecuteScript

This closes #443.

Signed-off-by: Andy LoPresto <alopresto@apache.org>


> Allow concurrent execution of ExecuteScript
> -------------------------------------------
>
>                 Key: NIFI-1822
>                 URL: https://issues.apache.org/jira/browse/NIFI-1822
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
>             Fix For: 1.0.0, 0.7.0
>
>
> Currently ExecuteScript is annotated as TriggerSerially, meaning only one task can be
running at a time. This causes issues for throughput, and can become a severe bottleneck in
a flow.
> Originally this was done because the bindings for the session, context, etc. were put
on a single script engine, so multiple tasks would clobber each other's bindings. However
as a tradeoff (memory for capability), it would be better to have a "pool" of Script Engine
instances, whose size is perhaps the max number of concurrent tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message