kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris <boris...@gmail.com>
Subject Re: Changing number of Kudu worker threads
Date Wed, 13 Feb 2019 02:45:18 GMT
Thanks Hao, appreciate your response.

Do we also need to bump other RPC thread related parameters queue etc.?

On Tue, Feb 12, 2019, 21:09 Hao Hao <hao.hao@cloudera.com wrote:

> Hi Boris,
>
> Sorry for the delay,  --rpc_num_service_threads sets the number of threads
>  in RPC service thread pool (the default is 20 for tablet server, 10 for
> master).  It should help with processing concurrent incoming RPC requests,
> but increasing it more than the number of available CPU cores of the
> machines may not bring too much value.
>
> You don't need to set the same value for masters and tablet servers. Most
> of the time, tablet servers should have more RPCs where the scans and
> writes are taking place.
>
> Best,
> Hao
>
> On Tue, Feb 12, 2019 at 5:29 PM Boris Tyukin <boris@boristyukin.com>
> wrote:
>
>> Can someone point us to documentation or explain what these parameters
>> really mean or how they should be set on production cluster?
>> I will greatly appreciate it!
>>
>> Boris
>>
>> On Fri, Feb 8, 2019 at 3:40 PM Boris Tyukin <boris@boristyukin.com>
>> wrote:
>>
>>> Hi guys,
>>>
>>> we need to process 1000s of operations per second and noticed that our
>>> Kudu 1.5 cluster was only using 10 threads while our application spins up
>>> 50 clients/threads. We observed in the web UI that only 10 threads are
>>> working and other 40 waiting in the queue.
>>>
>>> We found rpc_num_service_threads parameter in the configuration guide
>>> but it is still not clear to me what we need to adjust exactly to allow
>>> Kudu to handle more concurrent operations.
>>>
>>> Do we bump this parameter below or we need to consider other rpc related
>>> parameters?
>>>
>>> Also do we need to use the same numbers for Masters and tablets?
>>>
>>> Is there any good numbers to target based on CPU core count?
>>>
>>> --rpc_num_service_threads
>>> <https://kudu.apache.org/docs/configuration_reference.html#kudu-master_rpc_num_service_threads>
>>> <https://kudu.apache.org/docs/configuration_reference.html#kudu-master_rpc_num_service_threads>
>>>
>>> Number of RPC worker threads to run
>>>
>>> Type
>>>
>>> int32
>>>
>>> Default
>>>
>>> 10
>>>
>>> Tags
>>>
>>> advanced
>>>
>>

Mime
View raw message