kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Parallelisation factor in kafka streams
Date Wed, 06 Jul 2016 21:19:26 GMT
Jeyhun,

you cannot change the number of threads within an application instance,
but you can start new application instances. Internal Kafka Consumer
re-balance with re-assign the partitions over all running application
instances.

Not sure what you mean by "restart the cluster"? For sure, you do not
need to restart the Kafka Brokers. And for a Streams application, there
is no cluster. Streams applications are regular Java applications and
can run anywhere (not necessarily on the same machines as Kafka Brokers).

-Matthias

On 07/06/2016 10:33 PM, Jeyhun Karimov wrote:
> Thank you for your answer Matthias.
> Is it possible to change the parallelism in runtime? Or do we have to
> restart the cluster?
> 
> 
> On Wed, 6 Jul 2016 at 19:08, Matthias J. Sax <matthias@confluent.io> wrote:
> 
>> Hi Jeyhun,
>>
>> the number of partitions determine the number of tasks within a Kafka
>> Streams application and thus, the maximum number of parallelism for your
>> application.
>>
>> For more details see
>> http://docs.confluent.io/3.0.0/streams/architecture.html#parallelism-model
>>
>> You can set the number of threads for a single application instance, via
>> parameter "num.stream.threads" (default value is 1).
>>
>> See
>>
>> http://docs.confluent.io/3.0.0/streams/developer-guide.html#optional-configuration-parameters
>>
>>
>> -Matthias
>>
>> On 07/06/2016 06:11 PM, Jeyhun Karimov wrote:
>>> Hi community,
>>>
>>> How can I set parallelisation factor in kafka streams? Is it related with
>>> the number of partitions within topics?
>>>
>>>
>>> Cheers
>>> Jeyhun
>>>
>>
>> --
> -Cheers
> 
> Jeyhun
> 


Mime
View raw message