cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matija Gobec <matija0...@gmail.com>
Subject Re: Cqlsh timeout and schema refresh exceptions
Date Mon, 19 Dec 2016 21:02:48 GMT
There is an exposed API for schema agreement and I would advise you to use
that if you can.
Look at this JIRA ticket
<https://datastax-oss.atlassian.net/browse/JAVA-669>.

On Mon, Dec 19, 2016 at 8:46 PM, Vladimir Yudovin <vladyu@winguzone.com>
wrote:

> Regarding schema agreement - try to increase time between CF creation.
> Also stress-tool waits for schema, look on its code, probably it uses some
> methods to ensure schema distribution.
>
> Best regards, Vladimir Yudovin,
> *Winguzone <https://winguzone.com?from=list> - Cloud Cassandra Hosting*
>
>
> ---- On Mon, 19 Dec 2016 14:35:00 -0500 *Saumitra S
> <saumitra.srivastav7@gmail.com <saumitra.srivastav7@gmail.com>>* wrote
> ----
>
> Thanks Vladimir!
>
> Is there any known issue in 3.0.10, where creating "CF with large number
> of cols" or "creating large number of CFs quickly" one after other gives
> schema agreement issue?
>
> What other things can I try to support ~12000 CF without hitting schema
> agreement related issues? I can put more RAM and increase heap size(even if
> I need to spend time in GC tuning for such large heap), but the issue which
> I get with 2400 cols CFs starts happening just after few keyspaces(less
> than 200 CFs). What can I try to fix that?
>
>
>
>
>
> On Tue, Dec 20, 2016 at 12:53 AM, Vladimir Yudovin <vladyu@winguzone.com>
> wrote:
>
>
> >I want to dig deeper into what all things happen in C* at time of CF
> creation
>
> It starts somewhere in *MigrationManager.announceNewColumnFamily*
> function, I guess.
>
>
> >imitation of number of keyspaces which can be created.
>
> Actually it's CF limitation, not keyspaces.
>
>
> >if you can also point me to the this 1MB per CF thingy, it would be great.
>
> Look at http://www.mail-archive.com/user@cassandra.apache.org/
> msg46359.html, CASSANDRA-5935, CASSANDRA-2252
> In source look at *SlabAllocator.REGION_SIZE* definition.
>
>
> Best regards, Vladimir Yudovin,
> *Winguzone <https://winguzone.com?from=list> - Cloud Cassandra Hosting*
>
>
>
> ---- On Mon, 19 Dec 2016 14:10:37 -0500 *Saumitra S
> <saumitra.srivastav7@gmail.com <saumitra.srivastav7@gmail.com>>* wrote
> ----
>
> Hi Vladimir,
>
> Thanks for the response.
>
> When I see *"**com.datastax.driver.core.ControlConnection"* exceptions, I
> see that keyspaces and CF are created. But when I create CF with large
> number of columns(2400 cols) quickly one after the other(with 2 seconds gap
> between CREATE TABLE queries), I get schema agreement timeout errors (* com.datastax.driver.core.Cluster
> | Error while waiting for schema agreement). *This happens even with a
> clean slate(empty data directory), just after creating 4 keyspaces. Timeout
> is set to 30 seconds. Please note that CREATE TABLE queries are NOT fired
> in parallel. I wait for 1 query to complete(with schema agreement) before
> firing another one.
>
> I want to dig deeper into what all things happen in C* at time of CF
> creation to understand more about the limitation of number of keyspaces
> which can be created. Can you please point me to the corresponding source
> code? Specifically if you can also point me to the this 1MB per CF thingy,
> it would be great.
>
>
> Best Regards,
> Saumitra
>
>
>
>
>
>
>
>
>
> On Mon, Dec 19, 2016 at 11:41 PM, Vladimir Yudovin <vladyu@winguzone.com>
> wrote:
>
>
> Hi,
>
> *Question*: Does C* reads some schema/metadata on calling cqlsh, which is
> causing timeout with large number of keyspaces?
>
> A lot ). cqlsh reads schemas, cluster topology, each node tokens, etc. You
> can just capture TCP port 9042 (unless you use SSL)  and view all
> negotiation between cqlsh and node.
>
>
> *Question*: Can a single C* cluster of 5 nodes(32gb/8cpu each) support
> upto 500 keyspaces each having 25 CFs. What kind of issues I can expect?
>
> You have 500*25 = 12500 tables, it's huge number. Each CF takes at least
> 1M of heap memory. So it needs 12G heap only for starting usage. Make test
> on one-two node cluster.
>
>
> *Question*: What is the effect of below exception?
>
> Is keyspaces created despite exception or no?
>
> Best regards, Vladimir Yudovin,
> *Winguzone <https://winguzone.com?from=list> - Cloud Cassandra Hosting*
>
>
> ---- On Mon, 19 Dec 2016 10:24:20 -0500 *Saumitra S
> <saumitra.srivastav7@gmail.com <saumitra.srivastav7@gmail.com>>* wrote
> ----
>
> Hi All,
>
> I have a 2 node cluster(32gb ram/8cpu) running 3.0.10 and I created 50
> keyspaces in it. Each keyspace has 25 CF. Column count in each CF ranges
> between 5 to 30.
>
> I am getting few issues once keyspace count reaches ~50.
>
> *Issue 1:*
>
> When I try to use cqlsh, I get timeout.
>
> *$ cqlsh `hostname -i`*
> *Connection error: ('Unable to connect to any servers', {'10.0.20.220':
> OperationTimedOut('errors=None, last_host=None',)})*
>
> If I increase connect timeout, I am able to access cluster through cqlsh
>
> *$ cqlsh --connect-timeout 20  `hostname -i   //this works fine*
>
> *Question: *Does C* reads some schema/metadata on calling cqlsh, which is
> causing timeout with large number of keyspaces?
>
>
> *Issue 2:*
>
> If I create keyspaces which have 3 large CF(each having around 2500 cols),
> then I start to see schema agreement timeout in my logs. I have set schema
> agreement timeout to 30 seconds in driver.
>
> *2016-12-13 08:37:02.733 | gbd-std-01 | WARN | cluster2-worker-194 |
> com.datastax.driver.core.Cluster | Error while waiting for schema agreement*
>
> *Question:* Can a single C* cluster of 5 nodes(32gb/8cpu each) support
> upto 500 keyspaces each having 25 CFs. What kind of issues I can expect?
>
>
> *Issue 3:*
>
> I am creating keyspaces and CFs through datastax driver. I see following
> exception in my log after reaching *~50 keyspaces.*
>
> *Question: *What is the effect of below exception?
>
> 2016-12-19 13:55:35.615 | gbd-std-01 | ERROR | cluster1-worker-147 | *com.datastax.driver.core.ControlConnection
> | [Control connection] Unexpected error while refreshing schema*
> *java.util.concurrent.ExecutionException:
> com.datastax.driver.core.exceptions.OperationTimedOutException:
> [gbd-cass-20.ec2-east1.hidden.com/10.0.20.220
> <http://gbd-cass-20.ec2-east1.hidden.com/10.0.20.220>] Operation timed out*
>         at com.google.common.util.concurrent.AbstractFuture$
> Sync.getValue(AbstractFuture.java:299) ~[com.google.guava.guava-18.0.
> jar:na]
>         at com.google.common.util.concurrent.AbstractFuture$
> Sync.get(AbstractFuture.java:286) ~[com.google.guava.guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[com.google.guava.guava-18.0.jar:na]
>         at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.SchemaParser$V3SchemaParser.
> fetchSystemRows(SchemaParser.java:632) ~[com.datastax.cassandra.
> cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.Cluster$Manager$
> SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570)
> [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.ExceptionCatchingRunnable.run(
> ExceptionCatchingRunnable.java:32) [com.datastax.cassandra.
> cassandra-driver-core-3.0.0.jar:na]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException:
> [gbd-cass-20.ec2-east1.hidden.com/10.0.20.220] Operation timed out
>         at com.datastax.driver.core.DefaultResultSetFuture.onTimeout(
> DefaultResultSetFuture.java:209) ~[com.datastax.cassandra.
> cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1260)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581)
> ~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
>         at io.netty.util.HashedWheelTimer$HashedWheelBucket.
> expireTimeouts(HashedWheelTimer.java:655) ~[io.netty.netty-common-4.0.
> 33.Final.jar:4.0.33.Final]
>         at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367)
> ~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final]
>         ... 1 common frames omitted
> 2016-12-19 13:55:39.885 | gbd-std-01 | ERROR | cluster2-worker-124 | *com.datastax.driver.core.ControlConnection
> | [Control connection] Unexpected error while refreshing schema*
> *java.util.concurrent.ExecutionException:
> com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout
> during read query at consistency ONE (1 responses were required but only 0
> replica responded)*
>         at com.google.common.util.concurrent.AbstractFuture$
> Sync.getValue(AbstractFuture.java:299) ~[com.google.guava.guava-18.0.
> jar:na]
>         at com.google.common.util.concurrent.AbstractFuture$
> Sync.get(AbstractFuture.java:286) ~[com.google.guava.guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[com.google.guava.guava-18.0.jar:na]
>         at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.SchemaParser$V3SchemaParser.
> fetchSystemRows(SchemaParser.java:632) ~[com.datastax.cassandra.
> cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306)
> ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.Cluster$Manager$
> SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570)
> [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na]
>         at com.datastax.driver.core.ExceptionCatchingRunnable.run(
> ExceptionCatchingRunnable.java:32) [com.datastax.cassandra.
> cassandra-driver-core-3.0.0.jar:na]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_45]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_45]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
>
>
>
> Best Regards,
> Saumitra
>
>
>
>
>

Mime
View raw message