flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: threads, parallelism and task managers
Date Tue, 29 Mar 2016 09:51:58 GMT
Hey Stefano,

this should work by setting the parallelism on the environment, e.g.

env.setParallelism(32)

Is this what you are doing?

The task threads are not part of a pool, but each submitted task
creates its own Thread.

– Ufuk


On Fri, Mar 25, 2016 at 9:10 PM, Flavio Pompermaier
<pompermaier@okkam.it> wrote:
> Any help here? I think that the problem is that the JobManager creates the
> executionContext of the scheduler with
>
>        val executionContext = ExecutionContext.fromExecutor(new
> ForkJoinPool())
>
> and thus the number of concurrently running threads is limited to the number
> of cores (using the default constructor of the ForkJoinPool).
> What do you think?
>
>
> On Wed, Mar 23, 2016 at 6:55 PM, Stefano Bortoli <s.bortoli@gmail.com>
> wrote:
>>
>> Hi guys,
>>
>> I am trying to test a job that should run a number of tasks to read from a
>> RDBMS using an improved JDBC connector. The connection and the reading run
>> smoothly, but I cannot seem to be able to move above the limit of 8
>> concurrent threads running. 8 is of course the number of cores of my
>> machine.
>>
>> I have tried working around configurations and settings, but the Executor
>> within the ExecutionContext keeps on having a parallelism of 8. Although, of
>> course, the parallelism of the execution environment is much higher (in fact
>> I have many more tasks to be allocated).
>>
>> I feel it may be an issue of the LocalMiniCluster configuration that may
>> just override/neglect my wish for higher degree of parallelism. Is there a
>> way for me to work around this issue?
>>
>> please let me know. Thanks a lot for you help! :-)
>>
>> saluti,
>> Stefano
>
>
>

Mime
View raw message