cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ola Nowak <ola.nowa...@gmail.com>
Subject Random NoHostAvailableException using DataStax Java driver
Date Tue, 04 Nov 2014 11:45:02 GMT
Hi All :)
I have an application which use DataStax Java driver v2.0.2 to connect
to Cassandra Cluster (6 nodes, v 2.0.11).
Application is deployed in three copies on 3 different servers. From
time to time on random application server I get this exception:

2014-11-04 10:37:15,301 - ERROR: Servlet.service() for servlet [Unique
Identifier Service] in context with path [/uis] threw exception
[com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: [cassandra-11/10.0.0.11:9042,
/10.0.0.10:9042, /10.0.0.12:9042, /10.0.0.7:9042, /10.0.0.9:9042,
/10.0.0.8:9042] - use getErrors() for details)] with root cause
com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: [cassandra-11/10.0.0.11:9042,
/10.0.0.10:9042, /10.0.0.12:9042, /10.0.0.7:9042, /10.0.0.9:9042,
/10.0.0.8:9042] - use getErrors() for details)
        at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103)
        at com.datastax.driver.core.SessionManager.execute(SessionManager.java:418)
        at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:454)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:104)
        at com.datastax.driver.core.SessionManager.execute(SessionManager.java:92)
        at eu.europeana.cloud.service.uis.database.dao.CassandraDataProviderDAO.getProviders(CassandraDataProviderDAO.java:92)
        at eu.europeana.cloud.service.uis.CassandraDataProviderService.getProviders(CassandraDataProviderService.java:34)
        at eu.europeana.cloud.service.uis.rest.DataProvidersResource.getProviders(DataProvidersResource.java:54)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:353)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:343)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:983)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:359)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
2014-11-04 10:37:28,317 - ERROR: [Control connection] Cannot connect
to any host, scheduling retry

The error usually occurs on only one of the application nodes
(sometimes two) but application on other nodes is working fine.
I cannot see any errors in Cassandra logs, so it looks like if it was
working ok. Cassandra and application servers are in the same server
room with a high throughput network, so connection shouldn't be a
problem - and my admin claims that it is not. But even if there was an
error in connection the driver should retry to connect, right? (that's
what its documentation says and I'm using default connection
parameters)

I don't know where to look for the cause of the error. Is this a
matter of DataStax Driver or my code or rather Cassandra?
Any help appreciated.
Kind regards,
Ola

Mime
View raw message