cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Guyomar <nicolas.guyo...@gmail.com>
Subject Re: Error during select query - Found other issues with cluster too
Date Wed, 20 Dec 2017 09:18:19 GMT
Hi Dipan,

This seems like a really unbalanced modelisation, you have some very wide
rows !

Can you share your model and explain a bit what you are storing in this
table ? Your partition key might not be appropriate

On 20 December 2017 at 09:43, Dipan Shah <dipan.sha@hotmail.com> wrote:

> Hello Kurt,
>
>
> I think I might have found the problem:
>
>
> Can you please look at the tablehistogram for a table and see if that
> seems to be the problem? I think the Max Partition Size and Cell Count are
> too high:
>
>
> *Percentile* *SSTables* *Write Latency (micros)* *Read Latency (micros)* *Partition
> Size (bytes)* *Cell Count*
> 50.00% 0.00 0.00 0.00 29521 2299
> 75.00% 0.00 0.00 0.00 379022 29521
> 95.00% 0.00 0.00 0.00 5839588 454826
> 98.00% 0.00 0.00 0.00 30130992 2346799
> 99.00% 0.00 0.00 0.00 89970660 7007506
> Min 0.00 0.00 0.00 150 0
> Max 0.00 0.00 0.00 53142810146 1996099046
>
>
> Thanks,
>
> Dipan Shah
>
>
> ------------------------------
> *From:* Dipan Shah <dipan.sha@hotmail.com>
> *Sent:* Wednesday, December 20, 2017 12:04 PM
> *To:* User
> *Subject:* Re: Error during select query - Found other issues with
> cluster too
>
>
> Hello Kurt,
>
>
> We are using V 3.11.0 and I think this might a part of a bigger problem.
> I can see that nodes are failing in my cluster unexpectedly and also
> repair commands are failing.
>
>
> Repair command failure error:
>
>
> INFO  [Native-Transport-Requests-2] 2017-12-19 17:06:02,332
> Message.java:619 - Unexpected exception during request; channel = [id:
> 0xacc9a54a, L:/10.10.52.17:9042 ! R:/10.10.55.229:58712]
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> INFO  [Native-Transport-Requests-2] 2017-12-19 17:06:11,056
> Message.java:619 - Unexpected exception during request; channel = [id:
> 0xeebf628d, L:/10.10.52.17:9042 ! R:/10.10.55.229:58130]
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer
>
> Node failure error:
>
>
> ERROR [STREAM-IN-/10.10.52.22:7000] 2017-12-20 01:17:17,691
> JVMStabilityInspector.java:142 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.io.FileNotFoundException: /home/install/cassandra-3.11.
> 0/data/data/hhahistory/history-065e0c90d9be11e7afbcdfeb48785ac5/mc-19095-big-Filter.db
> (Too many open files)
> at java.io.FileOutputStream.open0(Native Method) ~[na:1.8.0_131]
> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> ~[na:1.8.0_131]
> at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
> ~[na:1.8.0_131]
> at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
> ~[na:1.8.0_131]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.
> flushBf(BigTableWriter.java:486) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.
> doPrepare(BigTableWriter.java:516) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.utils.concurrent.Transactional$
> AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter$
> TransactionalProxy.doPrepare(BigTableWriter.java:364)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.utils.concurrent.Transactional$
> AbstractTransactional.prepareToCommit(Transactional.java:173)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.utils.concurrent.Transactional$
> AbstractTransactional.finish(Transactional.java:184)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:264)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(
> SimpleSSTableMultiWriter.java:59) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.finish(
> RangeAwareSSTableWriter.java:129) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.streaming.StreamReceiveTask.
> received(StreamReceiveTask.java:110) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.streaming.StreamSession.
> receive(StreamSession.java:656) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.streaming.StreamSession.
> messageReceived(StreamSession.java:523) ~[apache-cassandra-3.11.0.jar:
> 3.11.0]
> at org.apache.cassandra.streaming.ConnectionHandler$
> IncomingMessageHandler.run(ConnectionHandler.java:317)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
>
>
>
> Thanks,
>
> Dipan Shah
>
>
> ------------------------------
> *From:* kurt greaves <kurt@instaclustr.com>
> *Sent:* Wednesday, December 20, 2017 2:23 AM
> *To:* User
> *Subject:* Re: Error during select query
>
> Can you send through the full stack trace as reported in the Cassandra
> logs? Also, what version are you running?
>
> On 19 Dec. 2017 9:23 pm, "Dipan Shah" <dipan.sha@hotmail.com> wrote:
>
> Hello,
>
>
> I am getting an error message when I'm running a select query from 1
> particular node. The error is "ServerError: java.lang.IllegalStateException:
> Unable to compute ceiling for max when histogram overflowed".
>
>
> Has anyone faced this error earlier? I tried to search for this but did
> not get anything that matches my scenario.
>
>
> Please note, I do not get this error when I run the same query from any
> other node. And I'm connecting to the node using cqlsh.
>
>
> Thanks,
>
> Dipan Shah
>
>

Mime
View raw message