cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cyril Scetbon <cyril.scet...@free.fr>
Subject Re: Hadoop + Yarn / DataStax driver issues
Date Tue, 24 Nov 2015 01:44:27 GMT
Similar tests with Spark and cassandra connector (which uses the native driver too) work well
... I was wondering if Yarn was restricting the network access but even executing spark (using
spark-shell) with --master yarn-client or --master yarn-server worked without any issue. You
could also see in the logs that it succeeded to connect to node006.internal.net

[cluster1-nio-worker-0]: DEBUG core.Connection: Connection[node006.internal.net/192.168.12.22:9042-1,
inFlight=0, closed=false] Connection opened successfully
[cluster1-nio-worker-1]: DEBUG Host.STATES: [node006.internal.net/192.168.12.22:9042] new
connection created, total = 2
then later got 

[node006.internal.net/192.168.12.22:9042] Pool is shutdown

However I've seen that before that, there is an error :

[cluster1-blocking-task-worker-0]: DEBUG Host.STATES: Defuncting Connection[node006.internal.net/192.168.12.22:9042-3,
inFlight=0, closed=true] because: [node006.internal.net/192.168.12.22:9042] Error while setting
keyspace

and after that error the node is marked as down :

DEBUG Host.STATES: [node006.internal.net/192.168.12.22:9042] marking host DOWN

still investigating ...

> On Nov 23, 2015, at 19:31, Patrick McFadin <pmcfadin@gmail.com> wrote:
> 
> That's a pretty basic error from the driver. I would look at the target node and make
sure that the IP/port combination is actually bound and free of any firewall shenanigans.

> 
> Patrick
> 
> On Mon, Nov 23, 2015 at 4:00 PM, Cyril Scetbon <cyril.scetbon@free.fr <mailto:cyril.scetbon@free.fr>>
wrote:
> Let me also add that I use Cassandra 2.1.11 with Hadoop 2.7.1 (HDP 2.3.2).
> 
> > Trying to execute an Hadoop job on Yarn, I got errors from Cassandra's internal
code. It seems that connections are closed but I can't understand why ...
> >
> > Here is a subtract of the errors :
> >
> > 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying node006.internal.net/192.168.12.22:9042
<http://node006.internal.net/192.168.12.22:9042>, trying next host (error is: com.datastax.driver.core.ConnectionException:
[node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>]
Pool is shutdown)
> > Failed with exception java.io.IOException:java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>
(com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>]
Pool is shutdown))
> > 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception java.io.IOException:java.io.IOException:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>
(com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>]
Pool is shutdown))
> > java.io.IOException: java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>
(com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>]
Pool is shutdown))
> >       at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> >       at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> >       at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> >       at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672)
> >       at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> >       at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> >       at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> >       at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> >       at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> >       at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:497)
> >       at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> >       at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> > Caused by: java.io.IOException: com.datastax.driver.core.exceptions.NoHostAvailableException:
All host(s) tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>
(com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>]
Pool is shutdown))
> >       at org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132)
> >       at org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
> >       at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
> >       at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> >       ... 15 more
> > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
tried
> > for
> > query failed (tried: node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>
(com.datastax.driver.core.ConnectionException: [node006.internal.net/192.168.12.22:9042 <http://node006.internal.net/192.168.12.22:9042>]
Pool is shutdown))
> >       at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
> >       at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
> >       at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214)
> >       at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)
> >       at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)
> >       at org.apache.cassandra.hadoop.cql3.CqlRecordReader.fetchKeys(CqlRecordReader.java:578)
> >       at org.apache.cassandra.hadoop.cql3.CqlRecordReader.buildQuery(CqlRecordReader.java:526)
> >       at org.apache.cassandra.hadoop.cql3.CqlRecordReader.initialize(CqlRecordReader.java:148)
I also add a file with the complete debug logs.
> >
> > You can also find the complete log at https://www.dropbox.com/s/thwugpw8amv5x9q/output.log?dl=0
<https://www.dropbox.com/s/thwugpw8amv5x9q/output.log?dl=0>
> 
> 


Mime
View raw message