kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Tyukin <bo...@boristyukin.com>
Subject Re: close Kudu client on timeout
Date Thu, 17 Jan 2019 21:44:36 GMT
Hi Alexey,

it was "single idle Kudu Java client that created so many threads". 20,000
threads in a few days to be precise :)  that code is running non-stop and
basically listens to kafka topics, then for every batch from kafka, we
create new kudu client instance, upsert data and close client.

the part we missed was *client.close()* in the end of that loop in the code
- once we put it in there, problem was solved.

So it is hard to tell if it was Java GC or something else.

But ideally, it would be nice, if Kudu server itself would kill idle
connections from clients on a timeout. I think Impala has similar global
setting.

--rpc_default_keepalive_time_ms  maybe it - I will look into this.

Thanks,
Boris

On Thu, Jan 17, 2019 at 2:51 PM Alexey Serbin <aserbin@cloudera.com> wrote:

> Hi Boris,
>
> Kudu servers have a setting for connection inactivity period: idle
> connections to the servers will be automatically closed after the specified
> time (--rpc_default_keepalive_time_ms is the flag).  So, from that
> perspective idle clients is not a big concern to the Kudu server side.
>
> As for your question, right now Kudu doesn't have a way to initiate a
> shutdown of an idle client from the server side.
>
> BTW, I'm curious what it was in your case you reported: were there too
> many idle Kudu client objects around created by the same application?  Or
> that was something else, like a single idle Kudu Java client that created
> so many threads?
>
>
> Thanks,
>
> Alexey
>
> On Wed, Jan 16, 2019 at 1:31 PM Boris Tyukin <boris@boristyukin.com>
> wrote:
>
>> sorry it is Java
>>
>> On Wed, Jan 16, 2019 at 3:32 PM Mike Percy <mpercy@apache.org> wrote:
>>
>>> Java or C++ / Python client?
>>>
>>> Mike
>>>
>>> Sent from my iPhone
>>>
>>> > On Jan 16, 2019, at 12:27 PM, Boris Tyukin <boris@boristyukin.com>
>>> wrote:
>>> >
>>> > Hi guys,
>>> >
>>> > is there a setting on Kudu server to close/clean-up inactive Kudu
>>> clients?
>>> >
>>> > we just found some rogue code that did not close client on code
>>> completion and wondering if we can prevent this in future on Kudu server
>>> level rather than relying on good developers.
>>> >
>>> > That code caused 22,000 threads opened on our edge node over the last
>>> few days.
>>> >
>>> > Boris
>>>
>>>

Mime
View raw message