cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mattias W (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11528) Server Crash when select returns more than a few hundred rows
Date Sat, 09 Apr 2016 13:42:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233551#comment-15233551
] 

Mattias W edited comment on CASSANDRA-11528 at 4/9/16 1:41 PM:
---------------------------------------------------------------

It is a out-of-memory error. Cassandra 3.4 on ubuntu 14.04 behaves the same, and there, the
last message in the log is below.

So now I know, select statements can use a lot of heap. The Ubuntu machine only has 1.5 GB
RAM. (The windows machine above had 8 GB)

{noformat}
INFO  [SharedPool-Worker-3] 2016-04-09 15:32:34,915 ApproximateTime.java:44 - Scheduling approximate
time-check task with a precision of 10 milliseconds
INFO  [Service Thread] 2016-04-09 15:34:50,366 GCInspector.java:284 - ConcurrentMarkSweep
GC in 443ms.  CMS Old Gen: 547965232 -> 268786192; Par Eden Space: 126017056 -> 0; Par
Survivor Space: 3420928 -> 0
INFO  [Service Thread] 2016-04-09 15:34:50,379 StatusLogger.java:52 - Pool Name          
         Active   Pending      Completed   Blocked  All Time Blocked
ERROR [SharedPool-Worker-2] 2016-04-09 15:34:50,409 JVMStabilityInspector.java:139 - JVM state
determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_77]
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77]
	at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:208) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:185)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:110)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:98)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:134)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1799)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2467)
~[apache-cassandra-3.4.jar:3.4]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_77]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.4.jar:3.4]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
INFO  [Service Thread] 2016-04-09 15:34:50,412 StatusLogger.java:56 - MutationStage      
              0         0            157         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,414 StatusLogger.java:56 - ViewMutationStage  
              0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,415 StatusLogger.java:56 - ReadStage          
              1         0             22         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,416 StatusLogger.java:56 - RequestResponseStage
             0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,418 StatusLogger.java:56 - ReadRepairStage    
              0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,423 StatusLogger.java:56 - CounterMutationStage
             0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,424 StatusLogger.java:56 - MiscStage          
              0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,428 StatusLogger.java:56 - CompactionExecutor 
              0         0             72         0                 0

{noformat}


was (Author: mattiasw2):
It is a out-of-memory error. Cassandra 3.4 on ubuntu 14.04 behaves the same, and there, the
last message in the log is

{noformat}
INFO  [SharedPool-Worker-3] 2016-04-09 15:32:34,915 ApproximateTime.java:44 - Scheduling approximate
time-check task with a precision of 10 milliseconds
INFO  [Service Thread] 2016-04-09 15:34:50,366 GCInspector.java:284 - ConcurrentMarkSweep
GC in 443ms.  CMS Old Gen: 547965232 -> 268786192; Par Eden Space: 126017056 -> 0; Par
Survivor Space: 3420928 -> 0
INFO  [Service Thread] 2016-04-09 15:34:50,379 StatusLogger.java:52 - Pool Name          
         Active   Pending      Completed   Blocked  All Time Blocked
ERROR [SharedPool-Worker-2] 2016-04-09 15:34:50,409 JVMStabilityInspector.java:139 - JVM state
determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Java heap space
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_77]
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77]
	at org.apache.cassandra.io.util.DataOutputBuffer.reallocate(DataOutputBuffer.java:126) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:86) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:374) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.Cell$Serializer.serialize(Cell.java:208) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:185)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:110)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:98)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:134)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) ~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1799)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2467)
~[apache-cassandra-3.4.jar:3.4]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_77]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.4.jar:3.4]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.4.jar:3.4]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
INFO  [Service Thread] 2016-04-09 15:34:50,412 StatusLogger.java:56 - MutationStage      
              0         0            157         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,414 StatusLogger.java:56 - ViewMutationStage  
              0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,415 StatusLogger.java:56 - ReadStage          
              1         0             22         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,416 StatusLogger.java:56 - RequestResponseStage
             0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,418 StatusLogger.java:56 - ReadRepairStage    
              0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,423 StatusLogger.java:56 - CounterMutationStage
             0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,424 StatusLogger.java:56 - MiscStage          
              0         0              0         0                 0

INFO  [Service Thread] 2016-04-09 15:34:50,428 StatusLogger.java:56 - CompactionExecutor 
              0         0             72         0                 0

{noformat}

> Server Crash when select returns more than a few hundred rows
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-11528
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11528
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: windows 7, 8 GB machine
>            Reporter: Mattias W
>             Fix For: 3.x
>
>         Attachments: datastax_ddc_server-stdout.2016-04-07.log
>
>
> While implementing a dump procedure, which did "select * from" from one table at a row,
I instantly kill the server. A simple 
> {noformat}select count(*) from {noformat} 
> also kills it. For a while, I thought the size of blobs were the cause
> I also try to only have a unique id as partition key, I was afraid a single partition
got too big or so, but that didn't change anything
> It happens every time, both from Java/Clojure and from DevCenter.
> I looked at the logs at C:\Program Files\DataStax-DDC\logs, but the crash is so quick,
so nothing is recorded there.
> There is a Java-out-of-memory in the logs, but that isn't from the time of the crash.
> It only happens for one table, it only has 15000 entries, but there are blobs and byte[]
stored there, size between 100kb - 4Mb. Total size for that table is about 6.5 GB on disk.
> I made a workaround by doing many small selects instead, each only fetching 100 rows.
> Is there a setting a can set to make the system log more eagerly, in order to at least
get a stacktrace or similar, that might help you.
> It is the prun_srv that dies. Restarting the NT service makes Cassandra run again



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message