impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Jacobs (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Date Wed, 23 Nov 2016 21:37:01 GMT
Matthew Jacobs has uploaded a new patch set (#2).

Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................

IMPALA-4522: Bound Kudu client threads to avoid stress crash

In stress testing on physical boxes (80 cores, 200gb ram) we
discovered that the Kudu Java client creates a huge number
of threads (2x the #cores) per Kudu client, and this was
causing the impalad to crash when the JVM couldn't create
more threads.

This addresses the issue by setting the number of Kudu
client worker threads rather than letting the Kudu client
pick the default (2 * #cores). The number set here was
suggested by the Kudu team as being sufficient for Impala's
FE usage and this has been tested for 8+ hours on the stress
cluster where the crash was previously observed quickly.

In the future, Impala should probably be sharing a single
Kudu client (it is multithreaded), but additional support
from Kudu may be needed to ensure this usage is correct
(e.g. client metadata may need invalidation after some
operations).

Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
---
M fe/src/main/java/org/apache/impala/util/KuduUtil.java
1 file changed, 17 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/5205/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj@cloudera.com>

Mime
View raw message