Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Subject: Re: Parallelisation factor in kafka streams
To: dev@kafka.apache.org
References: <CAPqqMpTkEDKssbYxsvLq0Gx8c4oKNJEb_Pjwonn16YU0TSdCXw@mail.gmail.com>
 <db59d64a-00c4-dff6-af59-4442453cbf52@confluent.io>
 <CAPqqMpSQv=JfQgg6pbbLPs7+WMc9Kt4Y1uJB=KQy=Mjunz4dxw@mail.gmail.com>
From: "Matthias J. Sax" <matthias@confluent.io>
Message-ID: <1cce0ffe-b26a-06a6-7ae3-2f5bfd4deb52@confluent.io>
Date: Wed, 6 Jul 2016 23:19:26 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Icedove/45.1.0
MIME-Version: 1.0
In-Reply-To: <CAPqqMpSQv=JfQgg6pbbLPs7+WMc9Kt4Y1uJB=KQy=Mjunz4dxw@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
 protocol="application/pgp-signature";
 boundary="uVkuSe4dREdu25cs0cFO9RVOf7gDrqBa4"
archived-at: Wed, 06 Jul 2016 21:22:15 -0000

--uVkuSe4dREdu25cs0cFO9RVOf7gDrqBa4
Content-Type: multipart/mixed; boundary="4hC8Ek52eIbgDSlVmnQqSJrDRDsrtBRtC"
From: "Matthias J. Sax" <matthias@confluent.io>
To: dev@kafka.apache.org
Message-ID: <1cce0ffe-b26a-06a6-7ae3-2f5bfd4deb52@confluent.io>
Subject: Re: Parallelisation factor in kafka streams
References: <CAPqqMpTkEDKssbYxsvLq0Gx8c4oKNJEb_Pjwonn16YU0TSdCXw@mail.gmail.com>
 <db59d64a-00c4-dff6-af59-4442453cbf52@confluent.io>
 <CAPqqMpSQv=JfQgg6pbbLPs7+WMc9Kt4Y1uJB=KQy=Mjunz4dxw@mail.gmail.com>
In-Reply-To: <CAPqqMpSQv=JfQgg6pbbLPs7+WMc9Kt4Y1uJB=KQy=Mjunz4dxw@mail.gmail.com>

--4hC8Ek52eIbgDSlVmnQqSJrDRDsrtBRtC
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Jeyhun,

you cannot change the number of threads within an application instance,
but you can start new application instances. Internal Kafka Consumer
re-balance with re-assign the partitions over all running application
instances.

Not sure what you mean by "restart the cluster"? For sure, you do not
need to restart the Kafka Brokers. And for a Streams application, there
is no cluster. Streams applications are regular Java applications and
can run anywhere (not necessarily on the same machines as Kafka Brokers).=


-Matthias

On 07/06/2016 10:33 PM, Jeyhun Karimov wrote:
> Thank you for your answer Matthias.
> Is it possible to change the parallelism in runtime? Or do we have to
> restart the cluster?
>=20
>=20
> On Wed, 6 Jul 2016 at 19:08, Matthias J. Sax <matthias@confluent.io> wr=
ote:
>=20
>> Hi Jeyhun,
>>
>> the number of partitions determine the number of tasks within a Kafka
>> Streams application and thus, the maximum number of parallelism for yo=
ur
>> application.
>>
>> For more details see
>> http://docs.confluent.io/3.0.0/streams/architecture.html#parallelism-m=
odel
>>
>> You can set the number of threads for a single application instance, v=
ia
>> parameter "num.stream.threads" (default value is 1).
>>
>> See
>>
>> http://docs.confluent.io/3.0.0/streams/developer-guide.html#optional-c=
onfiguration-parameters
>>
>>
>> -Matthias
>>
>> On 07/06/2016 06:11 PM, Jeyhun Karimov wrote:
>>> Hi community,
>>>
>>> How can I set parallelisation factor in kafka streams? Is it related =
with
>>> the number of partitions within topics?
>>>
>>>
>>> Cheers
>>> Jeyhun
>>>
>>
>> --
> -Cheers
>=20
> Jeyhun
>=20


--4hC8Ek52eIbgDSlVmnQqSJrDRDsrtBRtC--

--uVkuSe4dREdu25cs0cFO9RVOf7gDrqBa4
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJXfXXeAAoJECnhiMLycopPKoMP/17txcjweuWuKawwMK3Hezj8
fIEre2yLDbiP9BCvgUcJOTDvbbYUUilRwBff1drRa8jK+L5t9Z0lJtqv+N+C4m2J
HV5gvwem6ihgXEA8XmJDrDUBWCOqRR18575YLjV46EBUs+Tv9y9rrT1A5SnDS4oc
5Dd27k7imprCXHJLrTTdVh8lvzFJFGPDPn6/82izxv5ln8qUw/ssFfIE8N3LJ11Y
TuNZO16vIeZpNbHkHjFJ7I6HXHi7i2lnzgF9YtTjdDO5W2lTv6fqeYZHTloAamMv
TzU5h/neZLXrTpN4zHuOQ8Qkl6a/WwvgFXb+5apFszMRfaJqoII9VHlHyrTVoG+s
cNeWJMQ6ZoOaYDO1U9J8scMZ7wITpKe1Ei+8HoX5Y6POEvnAgqYqLyDspoat0FqQ
vb8f0sH8zpvHg6Ig+mtMG00Eq0BC58cBRLslUMwsJQxN+IJWtdXBy+InSPFXGWNl
DWtHoQpEfGKvByCKqeZyJUeV3rMJ5kOVYCdSJQHVDClAjRMFIA/u6QCo59oZe8nI
P2PtWOy+pF/EyFtXUDMTGMabS2aUoEM5z2gPeVzLM+rRbB0+DzUtKQgACypGOK85
PBsufaD/+VioQ3xh9zvFDzSF9xztl4oDShrTEXqLA8lpHdbH+6rKKu+h18G1f/9v
9vVyT0RTFYDMIpKkp6mK
=214R
-----END PGP SIGNATURE-----

--uVkuSe4dREdu25cs0cFO9RVOf7gDrqBa4--