cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Haddad <...@jonhaddad.com>
Subject Re: Solve Busy pool at Cassandra side
Date Sun, 13 May 2018 13:21:32 GMT
This error comes from com.datastax.driver.core.HostConnectionPool#enqueue,
which is the client side pool.  Cassandra can handle more requests, the
application needs to be fixed.

As per the java docs:

/**
 * Indicates that a connection pool has run out of available connections.
 * <p/>
 * This happens if the pool has no connections (for example if it's
currently reconnecting to its host), or if all
 * connections have reached their maximum number of in flight queries. The
query will be retried on the next host in the
 * {@link
com.datastax.driver.core.policies.LoadBalancingPolicy#newQueryPlan(String,
Statement) query plan}.
 * <p/>
 * This exception is a symptom that the driver is experiencing a high
workload. If it happens regularly on all hosts,
 * you should consider tuning one (or a combination of) the following
pooling options:
 * <ul>
 * <li>{@link
com.datastax.driver.core.PoolingOptions#setMaxRequestsPerConnection(HostDistance,
int)}: maximum number of
 * requests per connection;</li>
 * <li>{@link
com.datastax.driver.core.PoolingOptions#setMaxConnectionsPerHost(HostDistance,
int)}: maximum number of
 * connections in the pool;</li>
 * <li>{@link
com.datastax.driver.core.PoolingOptions#setMaxQueueSize(int)}: maximum
number of enqueued requests before
 * this exception is thrown.</li>
 * </ul>
 */

Using a third party lib is a bit of a pain, but with a little effort you
can modify the session no matter where it is.  It's possible to access and
modify private variables in the JVM, the Apache Commons library has a
module just for that:
https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/reflect/FieldUtils.html#readField-java.lang.reflect.Field-java.lang.Object-boolean-

I wrote an article with some detail here:
http://rustyrazorblade.com/post/2018/2018-02-25-accessing-private-variables-in-jvm/

If you're not sure where the session is, fire up your program with a
debugger and go look for it.  You can also find it using the Reflections
library (https://github.com/ronmamo/reflections) by
using getSubTypesOf(com.datastax.driver.core.Session).

Once you have the session you can bump up setMaxRequestsPerConnection.

This will allow you to circumvent the current issue, but you might run into
another - the client may be creating async requests without considering the
in flight queries, which isn't very smart.  Ideally it would either use a
Semaphore to limit the number of in flight or simply do SOME_SMALL_NUMBER
of queries at a time and wait for the ResultSetFuture to complete.  This is
in the FAQ of the driver:
https://github.com/datastax/java-driver/blob/3.x/faq/README.md#i-am-encountering-busypoolexception-what-does-this-mean-and-how-do-i-avoid-it

Hope this helps,
Jon


On Sun, May 13, 2018 at 8:29 AM onmstester onmstester <onmstester@zoho.com>
wrote:

> Hi,
> I'm getting "Pool is Busy (limit is 256)", while connecting to a single
> node cassandra cluster. The whole client side application is a 3rd-party lib
> which i can't change it's source and its session builder is not using any
> PoolingOptions.
> Is there any config on cassandra side that could handle increasing max
> requests per connections from 256 to 32K?
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>

Mime
View raw message