kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Hao <hao....@cloudera.com>
Subject Re: Changing number of Kudu worker threads
Date Wed, 13 Feb 2019 05:54:35 GMT
I don't see other flags that are relevant here, maybe others can chime in.

For --rpc_service_queue_length, it configs the size of the RPC request
queues. The queue helps to buffer requests in case if there is a bunch of
them coming at once and service threads are too busy processing already
arrived requests. But I don't see it can help with handling more concurrent
requests.

Best,
Hao

On Tue, Feb 12, 2019 at 6:45 PM Boris <boriskey@gmail.com> wrote:

> Thanks Hao, appreciate your response.
>
> Do we also need to bump other RPC thread related parameters queue etc.?
>
> On Tue, Feb 12, 2019, 21:09 Hao Hao <hao.hao@cloudera.com wrote:
>
>> Hi Boris,
>>
>> Sorry for the delay,  --rpc_num_service_threads sets the number of
>> threads in RPC service thread pool (the default is 20 for tablet server,
>> 10 for master).  It should help with processing concurrent incoming RPC
>> requests, but increasing it more than the number of available CPU cores of
>> the machines may not bring too much value.
>>
>> You don't need to set the same value for masters and tablet servers. Most
>> of the time, tablet servers should have more RPCs where the scans and
>> writes are taking place.
>>
>> Best,
>> Hao
>>
>> On Tue, Feb 12, 2019 at 5:29 PM Boris Tyukin <boris@boristyukin.com>
>> wrote:
>>
>>> Can someone point us to documentation or explain what these parameters
>>> really mean or how they should be set on production cluster?
>>> I will greatly appreciate it!
>>>
>>> Boris
>>>
>>> On Fri, Feb 8, 2019 at 3:40 PM Boris Tyukin <boris@boristyukin.com>
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> we need to process 1000s of operations per second and noticed that our
>>>> Kudu 1.5 cluster was only using 10 threads while our application spins up
>>>> 50 clients/threads. We observed in the web UI that only 10 threads are
>>>> working and other 40 waiting in the queue.
>>>>
>>>> We found rpc_num_service_threads parameter in the configuration guide
>>>> but it is still not clear to me what we need to adjust exactly to allow
>>>> Kudu to handle more concurrent operations.
>>>>
>>>> Do we bump this parameter below or we need to consider other
>>>> rpc related parameters?
>>>>
>>>> Also do we need to use the same numbers for Masters and tablets?
>>>>
>>>> Is there any good numbers to target based on CPU core count?
>>>>
>>>> --rpc_num_service_threads
>>>> <https://kudu.apache.org/docs/configuration_reference.html#kudu-master_rpc_num_service_threads>
>>>> <https://kudu.apache.org/docs/configuration_reference.html#kudu-master_rpc_num_service_threads>
>>>>
>>>> Number of RPC worker threads to run
>>>>
>>>> Type
>>>>
>>>> int32
>>>>
>>>> Default
>>>>
>>>> 10
>>>>
>>>> Tags
>>>>
>>>> advanced
>>>>
>>>

Mime
View raw message