kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adar Lieber-Dembo <a...@cloudera.com>
Subject Re: Kudu memory pressure
Date Tue, 04 Jun 2019 08:42:47 GMT
To add to Lifu's advice, assuming the transaction in this error is
representative of the average Kudu transaction, the tablet in question
has thousands of in-flight transactions. You should look into whether
one of its replicas is lagging and can't keep up with the incoming
writes.

On Tue, Jun 4, 2019 at 1:38 AM helifu <hzhelifu@corp.netease.com> wrote:
>
> Hi Dmitry,
>
>
>
> It seems there are many transactions are in flight. Maybe you can increase the transaction
memory limitation (default value is 64MB).
>
>
>
> Please look at below pages for detail:
>
> https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_tablet_transaction_memory_limit_mb
>
> https://kudu.apache.org/docs/command_line_tools_reference.html#tserver-set_flag
>
>
>
> And here is an example of increasing up to 128MB (please execute node-by-node):
>
> ./kudu tserver set_flag TSERVER-1 tablet_transaction_memory_limit_mb 128 –force
>
> ./kudu tserver set_flag TSERVER-2 tablet_transaction_memory_limit_mb 128 –force
>
> …
>
> ./kudu tserver set_flag TSERVER-n tablet_transaction_memory_limit_mb 128 --force
>
>
>
>
>
> 何李夫
>
> 2019-06-04 16:37:36
>
>
>
> 发件人: user-return-1654-hzhelifu=corp.netease.com@kudu.apache.org <user-return-1654-hzhelifu=corp.netease.com@kudu.apache.org>
代表 Дмитрий Павлов
> 发送时间: 2019年6月4日 16:10
> 收件人: user <user@kudu.apache.org>
> 主题: Kudu memory pressure
>
>
>
>
>
>
>
> Hi guys
>
>
>
> Hopes you are doing well!
>
>
>
> I have a question about Kudu and i will appreciate your reply
>
> During t-server log analyzing i have found huge number of following errors:
>
>
>
> W0530 02:15:10.793151    22 consensus_peers.cc:457] T 41e3be9529b74ed78dab8f1a9f3cda16
P 0203e2101faf406bba514b01e2bab246 -> Peer 4fc977ca8bc04d6cb855b0be09ad6f09 (tserver-4:7050):
Couldn't send request to peer 4fc977ca8bc04d6cb855b0be09ad6f09. Status: Illegal state: Rejecting
Update request from peer 0203e2101faf406bba514b01e2bab246 for term 20. Could not prepare a
single transaction due to: Service unavailable: transaction on tablet 41e3be9529b74ed78dab8f1a9f3cda16
rejected due to memory pressure: the memory usage of this transaction (16916) plus the current
consumption (67103079) exceeds the transaction memory limit (67108864) or the limit of an
ancestral memory tracker.. This is attempt 246: this message will repeat every 5th retry.
>
>
>
> Does it mean what Kudu t-server required additional memory space to process transactions
and default configuration options
>
> does not work correctly for our case?
>
>
>
> Are there any recommendations for us to fix this issue in terms of parameter tuning or
something else?
>
>
> Thanks Dmitry Pavlov

Mime
View raw message