incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Lejeune <d.leje...@pepite.be>
Subject outOfMemory error
Date Wed, 28 Nov 2012 13:06:50 GMT
Hi all,

I'm currently experiencing a outOfMemory problem with Cassandra-1.1.6 on
Windows XP-Pro (32-bit). The server crashes when I try to query it with a
relatively small amount of data (around 100 rows with 50000 columns each:
to be precise, on my configuration, querying 75 or more rows makes the
server to crash).
I tried with different library (Hector, JDBC, Thrift) and with the
Cassandra stress tool. All lead to the same outOfMemory problem.

My dataset is composed, for each row, of: 1 column in DateType, 49999
columns in DoubleType. I ran a query to fetch the entire dataset (around
330MB for the raw data + around 200MB for the metadata) and got the log at
the end of this message.

I also checked the heap-dump with Mat which displays these top values:
Class Name
             Objects                      Shallow Heap
java.nio.HeapByteBuffer                       16,253,559
780,170,832
bytes[]
16,254,013                 330,207,640     <-- Data ?
java.util.TreeMap$Entry                        8,126,711
260,054,752
org.apache.cassandra.db.Column         8,116,589
194,798,136     <-- Metadata ?

I tried to change the configuration in Cassandra for the values:
- row_cache_size_in_mb: tried different value between [0,1000] MB
- flush_largest_memtables_at: set to 0.1, but tried with 0.75
- reduce_cache_sizes_at: tried 0.85, 0.6, 0.2 and 0.1
- reduce_cache_capacity_to: tried 0.6 and 0.15
- memtable_total_space_in_mb: 64 MB, but also tried to disable it (-> 1/3
of the heap)
- Xms1G
- Xmx1500M
with no real observable improvements regarding my problem.

My Cassandra server and client both run on the same machine.

Here are the characteristics of my system configuration:
- Cassandra-1.1.6
- java version "1.6.0_20"
         Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
         Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)
- Windows XP-Pro 32 bits with service pack 3
- CPU double-core, 32 bits @2.26GHz
- 3.48 of RAM

I'm aware that my system configuration is not an optimized environment to
make Cassandra to run efficiently, but I wonder if you guys know a
workaround (or any idea on how) to fix this problem. Part of the answer is
probably that I do not have enough RAM to run the process, but I also
wonder if it is a 'normal' behaviour for Cassandra to handle this
particular test case that way.

Cheers,

Damien

---- Cassandra's LOG ---

Starting Cassandra Server
 INFO 09:10:27,171 Logging initialized
 INFO 09:10:27,171 JVM vendor/version: Java HotSpot(TM) Client VM/1.6.0_18
 INFO 09:10:27,171 Heap size: 1072103424/1569521664
 INFO 09:10:27,171 Classpath:
E:\dl_benchmark\db\apache-cassandra-1.1.6\conf;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\antlr-3.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-clientutil-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-thrift-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\avro-1.4.0-fixes.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\avro-1.4.0-sources-fixes.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-cli-1.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-codec-1.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-lang-2.4.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\compress-lzf-0.8.4.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\concurrentlinkedhashmap-lru-1.3.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\guava-r08.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\high-scale-lib-1.1.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jackson-core-asl-1.9.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jackson-mapper-asl-1.9.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jamm-0.2.5.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jline-0.9.94.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\json-simple-1.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\libthrift-0.7.0.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\log4j-1.2.16.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\metrics-core-2.0.3.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\servlet-api-2.5-20081211.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\slf4j-api-1.6.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\slf4j-log4j12-1.6.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snakeyaml-1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snappy-java-1.0.4.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snaptree-0.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\build\classes\main;E:\dl_benchmark\db\apache-cassandra-1.1.6\build\classes\thrift;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jamm-0.2.5.jar
 INFO 09:10:27,171 JNA not found. Native methods will be disabled.
 INFO 09:10:27,187 Loading settings from
file:/E:/dl_benchmark/db/apache-cassandra-1.1.6/conf/cassandra.yaml
 INFO 09:10:27,296 32bit JVM detected.  It is recommended to run Cassandra
on a 64bit JVM for better performance.
 INFO 09:10:27,296 DiskAccessMode 'auto' determined to be standard,
indexAccessMode is standard
 INFO 09:10:27,500 Global memtable threshold is enabled at 64MB
 INFO 09:10:27,781 Initializing key cache with capacity of 51 MBs.
 INFO 09:10:27,812 Scheduling key cache save to each 14400 seconds (going
to save all keys).
 INFO 09:10:27,812 Initializing row cache with capacity of 1000 MBs and
provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:10:27,812 Scheduling row cache save to each 0 seconds (going to
save all keys).
 INFO 09:10:27,906 reading saved cache
\var\lib\cassandra\saved_caches\system-schema_columnfamilies-KeyCache
 INFO 09:10:27,921 Opening
\var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-hf-134
(4938 bytes)
 INFO 09:10:27,921 Opening
\var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-hf-133
(5122 bytes)
 INFO 09:10:27,937 reading saved cache
\var\lib\cassandra\saved_caches\system-schema_columns-KeyCache
 INFO 09:10:27,937 Opening
\var\lib\cassandra\data\system\schema_columns\system-schema_columns-hf-74
(59 bytes)
 INFO 09:10:27,937 Opening
\var\lib\cassandra\data\system\schema_columns\system-schema_columns-hf-73
(13836053 bytes)
 INFO 09:10:27,953 Opening
\var\lib\cassandra\data\system\Versions\system-Versions-hf-64 (247 bytes)
 INFO 09:10:27,953 Opening
\var\lib\cassandra\data\system\schema_keyspaces\system-schema_keyspaces-hf-105
(1244 bytes)
 INFO 09:10:27,953 reading saved cache
\var\lib\cassandra\saved_caches\system-LocationInfo-KeyCache
 INFO 09:10:27,953 Opening
\var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-134 (346
bytes)
 INFO 09:10:32,843 reading saved cache
\var\lib\cassandra\saved_caches\Keyspace1-Standard1-KeyCache
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-552
(83124150 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-44
(5541610 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-592
(5541610 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-591
(5541610 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-555
(138540250 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-565
(60957710 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-484
(138540250 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-586
(49874490 bytes)
 INFO 09:10:32,859 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-437
(138540250 bytes)
 INFO 09:10:32,875 Opening
\var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-590
(60957710 bytes)
 INFO 09:10:32,875 Opening
\var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-133 (5514794
bytes)
 INFO 09:10:32,890 Opening
\var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-130 (195755351
bytes)
 INFO 09:10:32,906 Opening
\var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-131 (201295520
bytes)
 INFO 09:10:32,906 Opening
\var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-132 (30327963
bytes)
 INFO 09:10:32,906 Opening
\var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-129 (336374894
bytes)
 INFO 09:10:32,953 Opening
\var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-198 (91008900 bytes)
 INFO 09:10:32,968 Opening
\var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-202 (57927906 bytes)
 INFO 09:10:32,968 Opening
\var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-188 (2917363977
bytes)
 INFO 09:10:33,156 Opening \var\lib\cassandra\data\DEMO\Row\DEMO-Row-hf-1
(100 bytes)
 INFO 09:10:33,171 Opening
\var\lib\cassandra\data\TESTBENCHTHREE\Row\TESTBENCHTHREE-Row-hf-60
(96519635 bytes)
 INFO 09:10:33,171 Opening
\var\lib\cassandra\data\TESTBENCHTHREE\Row\TESTBENCHTHREE-Row-hf-72
(63929481 bytes)
 INFO 09:10:43,890 completed pre-loading (10 keys) key cache.
 INFO 09:10:44,000 Replaying
\var\lib\cassandra\commitlog\CommitLog-1354090021204.log,
\var\lib\cassandra\commitlog\CommitLog-1354090021205.log
 INFO 09:10:44,015 Replaying
\var\lib\cassandra\commitlog\CommitLog-1354090021204.log
 INFO 09:10:44,031 Finished reading
\var\lib\cassandra\commitlog\CommitLog-1354090021204.log
 INFO 09:10:44,031 Replaying
\var\lib\cassandra\commitlog\CommitLog-1354090021205.log
 INFO 09:10:44,031 Finished reading
\var\lib\cassandra\commitlog\CommitLog-1354090021205.log
 INFO 09:10:44,046 Enqueuing flush of Memtable-Versions@6172767(83/103
serialized/live bytes, 3 ops)
 INFO 09:10:44,062 Writing Memtable-Versions@6172767(83/103 serialized/live
bytes, 3 ops)
 INFO 09:10:44,296 Completed flushing
\var\lib\cassandra\data\system\Versions\system-Versions-hf-65-Data.db (247
bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
position=0)
 INFO 09:10:44,312 Log replay complete, 3 replayed mutations
 INFO 09:10:44,890 GC for ConcurrentMarkSweep: 210 ms for 1 collections,
41139480 used; max is 1569521664
 INFO 09:10:49,906 Cassandra version: 1.1.6
 INFO 09:10:49,906 Thrift API version: 19.32.0
 INFO 09:10:49,906 CQL supported versions: 2.0.0,3.0.0-beta1 (default:
2.0.0)
 INFO 09:10:49,953 Loading persisted ring state
 INFO 09:10:49,953 Starting up server gossip
 INFO 09:10:49,953 Enqueuing flush of Memtable-LocationInfo@8678192(29/36
serialized/live bytes, 1 ops)
 INFO 09:10:49,968 Writing Memtable-LocationInfo@8678192(29/36
serialized/live bytes, 1 ops)
 INFO 09:10:49,968 Completed flushing
\var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-135-Data.db
(80 bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
position=363)
 INFO 09:10:55,890 Starting Messaging Service on port 7000
 INFO 09:10:55,890 Using saved token 6783448253301637730960311964445940653
 INFO 09:10:55,906 Enqueuing flush of Memtable-LocationInfo@24013134(53/66
serialized/live bytes, 2 ops)
 INFO 09:10:55,906 Writing Memtable-LocationInfo@24013134(53/66
serialized/live bytes, 2 ops)
 INFO 09:10:55,906 Completed flushing
\var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-136-Data.db
(163 bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
position=544)
 INFO 09:10:55,906 Node localhost/127.0.0.1 state jump to normal
 INFO 09:10:55,921 Bootstrap/Replace/Move completed! Now serving reads.
 INFO 09:10:55,953 Binding thrift service to localhost/127.0.0.1:9160
 INFO 09:10:55,953 Using TFastFramedTransport with a max frame size of
33554432 bytes.
 INFO 09:10:55,953 Using synchronous/threadpool thrift server on localhost/
127.0.0.1 : 9160
 INFO 09:10:55,953 Listening for thrift clients...
 --- The Querying starts here  ---
 INFO 09:14:33,375 GC for ParNew: 742 ms for 1 collections, 1529127672
used; max is 1569521664
 INFO 09:14:33,375 GC for ConcurrentMarkSweep: 7691 ms for 1 collections,
1529135040 used; max is 1569521664
 INFO 09:14:33,421 Pool Name                    Active   Pending   Blocked
 INFO 09:14:33,421 ReadStage                         0         0         0
 INFO 09:14:33,421 RequestResponseStage              0         0         0
 INFO 09:14:33,421 ReadRepairStage                   0         0         0
 INFO 09:14:33,421 MutationStage                     0         0         0
 INFO 09:14:33,421 ReplicateOnWriteStage             0         0         0
 INFO 09:14:33,421 GossipStage                       0         0         0
 INFO 09:14:33,421 AntiEntropyStage                  0         0         0
 INFO 09:14:33,421 MigrationStage                    0         0         0
 INFO 09:14:33,421 StreamStage                       0         0         0
 INFO 09:14:33,421 MemtablePostFlusher               0         0         0
 INFO 09:14:33,421 FlushWriter                       0         0         0
 INFO 09:14:33,421 MiscStage                         0         0         0
 INFO 09:14:33,421 commitlog_archiver                0         0         0
 INFO 09:14:33,421 InternalResponseStage             0         0         0
 INFO 09:14:33,421 HintedHandoff                     0         0         0
 INFO 09:14:33,421 CompactionManager                 0         0
 INFO 09:14:33,468 MessagingService                n/a       0,0
 INFO 09:14:33,468 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:14:33,468 KeyCache                         14
1114112
all
 INFO 09:14:33,468 RowCache                          0
1048576000                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:14:33,468 ColumnFamily                Memtable ops,data
 INFO 09:14:33,468 system.NodeIdInfo                         0,0
 INFO 09:14:33,468 system.IndexInfo                          0,0
 INFO 09:14:33,468 system.LocationInfo                       0,0
 INFO 09:14:33,468 system.Versions                         3,103
 INFO 09:14:33,468 system.schema_keyspaces                   0,0
 INFO 09:14:33,468 system.Migrations                         0,0
 INFO 09:14:33,468 system.schema_columnfamilies                 0,0
 INFO 09:14:33,468 system.schema_columns                     0,0
 INFO 09:14:33,468 system.HintsColumnFamily                  0,0
 INFO 09:14:33,468 system.Schema                             0,0
 INFO 09:14:33,468 Keyspace1.SuperCounter1                   0,0
 INFO 09:14:33,468 Keyspace1.Standard1                       0,0
 INFO 09:14:33,468 Keyspace1.Super1                          0,0
 INFO 09:14:33,468 Keyspace1.Counter1                        0,0
 INFO 09:14:33,468 TESTBENCHTWO.Row                          0,0
 INFO 09:14:33,468 TESTBENCH.Row                             0,0
 INFO 09:14:33,468 DEMO.Row                                  0,0
 INFO 09:14:33,468 TESTBENCHTHREE.Row                        0,0
 WARN 09:14:33,468 Heap is 0.9742681958928348 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra is now reducing cache sizes to free
up memory.  Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you
don't want Cassandra to do this automatically
 WARN 09:14:33,468 Reducing RowCache capacity from 1048576000 to 0 to
reduce memory pressure
 WARN 09:14:33,468 Reducing KeyCache capacity from 1114112 to 8 to reduce
memory pressure
 WARN 09:14:33,468 Heap is 0.9742681958928348 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
 WARN 09:14:33,468 Flushing CFS(Keyspace='system', ColumnFamily='Versions')
to relieve memory pressure
 INFO 09:14:33,468 Enqueuing flush of Memtable-Versions@9422607(83/103
serialized/live bytes, 3 ops)
 INFO 09:14:33,468 Writing Memtable-Versions@9422607(83/103 serialized/live
bytes, 3 ops)
 INFO 09:14:33,562 Completed flushing
\var\lib\cassandra\data\system\Versions\system-Versions-hf-66-Data.db (247
bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
position=544)
 INFO 09:14:42,968 GC for ConcurrentMarkSweep: 8900 ms for 1 collections,
1549381648 used; max is 1569521664
 INFO 09:14:42,968 Pool Name                    Active   Pending   Blocked
 INFO 09:14:42,968 ReadStage                         0         0         0
 INFO 09:14:42,968 RequestResponseStage              0         0         0
 INFO 09:14:42,984 ReadRepairStage                   0         0         0
 INFO 09:14:42,984 MutationStage                     0         0         0
 INFO 09:14:42,984 ReplicateOnWriteStage             0         0         0
 INFO 09:14:43,015 GossipStage                       0         0         0
 INFO 09:14:43,015 AntiEntropyStage                  0         0         0
 INFO 09:14:43,015 MigrationStage                    0         0         0
 INFO 09:14:43,015 StreamStage                       0         0         0
 INFO 09:14:43,015 MemtablePostFlusher               0         0         0
 INFO 09:14:43,015 FlushWriter                       0         0         0
 INFO 09:14:43,015 MiscStage                         0         0         0
 INFO 09:14:43,015 commitlog_archiver                0         0         0
 INFO 09:14:43,015 InternalResponseStage             0         0         0
 INFO 09:14:43,015 HintedHandoff                     0         0         0
 INFO 09:14:43,015 CompactionManager                 0         0
 INFO 09:14:43,015 MessagingService                n/a       0,0
 INFO 09:14:43,015 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:14:43,015 KeyCache
8                        8
all
 INFO 09:14:43,015 RowCache
0                        0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:14:43,015 ColumnFamily                Memtable ops,data
 INFO 09:14:43,015 system.NodeIdInfo                         0,0
 INFO 09:14:43,015 system.IndexInfo                          0,0
 INFO 09:14:43,015 system.LocationInfo                       0,0
 INFO 09:14:43,015 system.Versions                           0,0
 INFO 09:14:43,015 system.schema_keyspaces                   0,0
 INFO 09:14:43,015 system.Migrations                         0,0
 INFO 09:14:43,015 system.schema_columnfamilies                 0,0
 INFO 09:14:43,015 system.schema_columns                     0,0
 INFO 09:14:43,015 system.HintsColumnFamily                  0,0
 INFO 09:14:43,015 system.Schema                             0,0
 INFO 09:14:43,015 Keyspace1.SuperCounter1                   0,0
 INFO 09:14:43,015 Keyspace1.Standard1                       0,0
 INFO 09:14:43,015 Keyspace1.Super1                          0,0
 INFO 09:14:43,015 Keyspace1.Counter1                        0,0
 INFO 09:14:43,015 TESTBENCHTWO.Row                          0,0
 INFO 09:14:43,015 TESTBENCH.Row                             0,0
 INFO 09:14:43,015 DEMO.Row                                  0,0
 INFO 09:14:43,015 TESTBENCHTHREE.Row                        0,0
 WARN 09:14:43,015 Heap is 0.9871680547889525 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
 INFO 09:14:43,015 Unable to reduce heap usage since there are no dirty
column families
 INFO 09:14:50,046 GC for ConcurrentMarkSweep: 6530 ms for 1 collections,
1561411240 used; max is 1569521664
 INFO 09:14:50,234 Pool Name                    Active   Pending   Blocked
 INFO 09:14:50,234 ReadStage                         0         0         0
 INFO 09:14:50,234 RequestResponseStage              0         0         0
 INFO 09:14:50,234 ReadRepairStage                   0         0         0
 INFO 09:14:50,234 MutationStage                     0         0         0
 INFO 09:14:50,234 ReplicateOnWriteStage             0         0         0
 INFO 09:14:50,234 GossipStage                       0         0         0
 INFO 09:14:50,234 AntiEntropyStage                  0         0         0
 INFO 09:14:50,234 MigrationStage                    0         0         0
 INFO 09:14:50,234 StreamStage                       0         0         0
 INFO 09:14:50,234 MemtablePostFlusher               0         0         0
 INFO 09:14:50,234 FlushWriter                       0         0         0
 INFO 09:14:50,234 MiscStage                         0         0         0
 INFO 09:14:50,234 commitlog_archiver                0         0         0
 INFO 09:14:50,234 InternalResponseStage             0         0         0
 INFO 09:14:50,234 HintedHandoff                     0         0         0
 INFO 09:14:50,234 CompactionManager                 0         0
 INFO 09:14:50,234 MessagingService                n/a       0,0
 INFO 09:14:50,234 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:14:50,234 KeyCache
8                        8
all
 INFO 09:14:50,234 RowCache
0                        0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:14:50,234 ColumnFamily                Memtable ops,data
 INFO 09:14:50,234 system.NodeIdInfo                         0,0
 INFO 09:14:50,234 system.IndexInfo                          0,0
 INFO 09:14:50,234 system.LocationInfo                       0,0
 INFO 09:14:50,234 system.Versions                           0,0
 INFO 09:14:50,234 system.schema_keyspaces                   0,0
 INFO 09:14:50,234 system.Migrations                         0,0
 INFO 09:14:50,234 system.schema_columnfamilies                 0,0
 INFO 09:14:50,234 system.schema_columns                     0,0
 INFO 09:14:50,265 system.HintsColumnFamily                  0,0
 INFO 09:14:50,265 system.Schema                             0,0
 INFO 09:14:50,265 Keyspace1.SuperCounter1                   0,0
 INFO 09:14:50,265 Keyspace1.Standard1                       0,0
 INFO 09:14:50,265 Keyspace1.Super1                          0,0
 INFO 09:14:50,265 Keyspace1.Counter1                        0,0
 INFO 09:14:50,265 TESTBENCHTWO.Row                          0,0
 INFO 09:14:50,265 TESTBENCH.Row                             0,0
 INFO 09:14:50,265 DEMO.Row                                  0,0
 INFO 09:14:50,265 TESTBENCHTHREE.Row                        0,0
 WARN 09:14:50,265 Heap is 0.9948325504604185 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
 INFO 09:14:50,265 Unable to reduce heap usage since there are no dirty
column families
 INFO 09:14:59,343 GC for ConcurrentMarkSweep: 8724 ms for 1 collections,
1566184304 used; max is 1569521664
 INFO 09:14:59,343 Pool Name                    Active   Pending   Blocked
 INFO 09:14:59,343 ReadStage                         0         0         0
 INFO 09:14:59,343 RequestResponseStage              0         0         0
 INFO 09:14:59,343 ReadRepairStage                   0         0         0
 INFO 09:14:59,343 MutationStage                     0         0         0
 INFO 09:14:59,343 ReplicateOnWriteStage             0         0         0
 INFO 09:14:59,343 GossipStage                       0         0         0
 INFO 09:14:59,343 AntiEntropyStage                  0         0         0
 INFO 09:14:59,343 MigrationStage                    0         0         0
 INFO 09:14:59,343 StreamStage                       0         0         0
 INFO 09:14:59,343 MemtablePostFlusher               0         0         0
 INFO 09:14:59,343 FlushWriter                       0         0         0
 INFO 09:14:59,343 MiscStage                         0         0         0
 INFO 09:14:59,343 commitlog_archiver                0         0         0
 INFO 09:14:59,343 InternalResponseStage             0         0         0
 INFO 09:14:59,343 HintedHandoff                     0         0         0
 INFO 09:14:59,343 CompactionManager                 0         0
 INFO 09:14:59,343 MessagingService                n/a       0,0
 INFO 09:14:59,343 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:14:59,343 KeyCache
8                        8
all
 INFO 09:14:59,343 RowCache
0                        0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:14:59,343 ColumnFamily                Memtable ops,data
 INFO 09:14:59,343 system.NodeIdInfo                         0,0
 INFO 09:14:59,343 system.IndexInfo                          0,0
 INFO 09:14:59,343 system.LocationInfo                       0,0
 INFO 09:14:59,343 system.Versions                           0,0
 INFO 09:14:59,343 system.schema_keyspaces                   0,0
 INFO 09:14:59,343 system.Migrations                         0,0
 INFO 09:14:59,343 system.schema_columnfamilies                 0,0
 INFO 09:14:59,343 system.schema_columns                     0,0
 INFO 09:14:59,343 system.HintsColumnFamily                  0,0
 INFO 09:14:59,343 system.Schema                             0,0
 INFO 09:14:59,343 Keyspace1.SuperCounter1                   0,0
 INFO 09:14:59,343 Keyspace1.Standard1                       0,0
 INFO 09:14:59,343 Keyspace1.Super1                          0,0
 INFO 09:14:59,343 Keyspace1.Counter1                        0,0
 INFO 09:14:59,343 TESTBENCHTWO.Row                          0,0
 INFO 09:14:59,343 TESTBENCH.Row                             0,0
 INFO 09:14:59,343 DEMO.Row                                  0,0
 INFO 09:14:59,343 TESTBENCHTHREE.Row                        0,0
 WARN 09:14:59,343 Heap is 0.997873645151546 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
 INFO 09:14:59,343 Unable to reduce heap usage since there are no dirty
column families
 INFO 09:15:08,500 GC for ConcurrentMarkSweep: 8854 ms for 1 collections,
1567778080 used; max is 1569521664
 INFO 09:15:08,500 Pool Name                    Active   Pending   Blocked
 INFO 09:15:08,500 ReadStage                         0         0         0
 INFO 09:15:08,500 RequestResponseStage              0         0         0
 INFO 09:15:08,500 ReadRepairStage                   0         0         0
 INFO 09:15:08,500 MutationStage                     0         0         0
 INFO 09:15:08,500 ReplicateOnWriteStage             0         0         0
 INFO 09:15:08,500 GossipStage                       0         0         0
 INFO 09:15:08,515 AntiEntropyStage                  0         0         0
 INFO 09:15:08,515 MigrationStage                    0         0         0
 INFO 09:15:08,515 StreamStage                       0         0         0
 INFO 09:15:08,515 MemtablePostFlusher               0         0         0
 INFO 09:15:08,515 FlushWriter                       0         0         0
 INFO 09:15:08,515 MiscStage                         0         0         0
 INFO 09:15:08,515 commitlog_archiver                0         0         0
 INFO 09:15:08,515 InternalResponseStage             0         0         0
 INFO 09:15:08,515 HintedHandoff                     0         0         0
 INFO 09:15:08,515 CompactionManager                 0         0
 INFO 09:15:08,515 MessagingService                n/a       0,0
 INFO 09:15:08,515 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:15:08,515 KeyCache
8                        8
all
 INFO 09:15:08,515 RowCache
0                        0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:15:08,515 ColumnFamily                Memtable ops,data
 INFO 09:15:08,515 system.NodeIdInfo                         0,0
 INFO 09:15:08,515 system.IndexInfo                          0,0
 INFO 09:15:08,515 system.LocationInfo                       0,0
 INFO 09:15:08,515 system.Versions                           0,0
 INFO 09:15:08,515 system.schema_keyspaces                   0,0
 INFO 09:15:08,515 system.Migrations                         0,0
 INFO 09:15:08,515 system.schema_columnfamilies                 0,0
 INFO 09:15:08,515 system.schema_columns                     0,0
 INFO 09:15:08,515 system.HintsColumnFamily                  0,0
 INFO 09:15:08,515 system.Schema                             0,0
 INFO 09:15:08,515 Keyspace1.SuperCounter1                   0,0
 INFO 09:15:08,515 Keyspace1.Standard1                       0,0
 INFO 09:15:08,515 Keyspace1.Super1                          0,0
 INFO 09:15:08,515 Keyspace1.Counter1                        0,0
 INFO 09:15:08,515 TESTBENCHTWO.Row                          0,0
 INFO 09:15:08,515 TESTBENCH.Row                             0,0
 INFO 09:15:08,515 DEMO.Row                                  0,0
 INFO 09:15:08,515 TESTBENCHTHREE.Row                        0,0
 WARN 09:15:08,515 Heap is 0.9988890984814084 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
 INFO 09:15:08,515 Unable to reduce heap usage since there are no dirty
column families
 INFO 09:15:13,734 GC for ConcurrentMarkSweep: 5129 ms for 1 collections,
1568572088 used; max is 1569521664
 INFO 09:15:13,734 Pool Name                    Active   Pending   Blocked
 INFO 09:15:13,734 ReadStage                         0         0         0
 INFO 09:15:13,734 RequestResponseStage              0         0         0
 INFO 09:15:13,734 ReadRepairStage                   0         0         0
 INFO 09:15:13,734 MutationStage                     0         0         0
 INFO 09:15:13,734 ReplicateOnWriteStage             0         0         0
 INFO 09:15:13,734 GossipStage                       0         0         0
 INFO 09:15:13,734 AntiEntropyStage                  0         0         0
 INFO 09:15:13,734 MigrationStage                    0         0         0
 INFO 09:15:13,734 StreamStage                       0         0         0
 INFO 09:15:13,734 MemtablePostFlusher               0         0         0
 INFO 09:15:13,734 FlushWriter                       0         0         0
 INFO 09:15:13,734 MiscStage                         0         0         0
 INFO 09:15:13,765 commitlog_archiver                0         0         0
 INFO 09:15:13,765 InternalResponseStage             0         0         0
 INFO 09:15:13,765 HintedHandoff                     0         0         0
 INFO 09:15:13,765 CompactionManager                 0         0
 INFO 09:15:13,765 MessagingService                n/a       0,0
 INFO 09:15:13,765 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:15:13,765 KeyCache
8                        8
all
 INFO 09:15:13,765 RowCache
0                        0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:15:13,765 ColumnFamily                Memtable ops,data
 INFO 09:15:13,765 system.NodeIdInfo                         0,0
 INFO 09:15:13,765 system.IndexInfo                          0,0
 INFO 09:15:13,765 system.LocationInfo                       0,0
 INFO 09:15:13,765 system.Versions                           0,0
 INFO 09:15:13,765 system.schema_keyspaces                   0,0
 INFO 09:15:13,765 system.Migrations                         0,0
 INFO 09:15:13,765 system.schema_columnfamilies                 0,0
 INFO 09:15:13,765 system.schema_columns                     0,0
 INFO 09:15:13,765 system.HintsColumnFamily                  0,0
 INFO 09:15:13,765 system.Schema                             0,0
 INFO 09:15:13,765 Keyspace1.SuperCounter1                   0,0
 INFO 09:15:13,765 Keyspace1.Standard1                       0,0
 INFO 09:15:13,765 Keyspace1.Super1                          0,0
 INFO 09:15:13,765 Keyspace1.Counter1                        0,0
 INFO 09:15:13,765 TESTBENCHTWO.Row                          0,0
 INFO 09:15:13,765 TESTBENCH.Row                             0,0
 INFO 09:15:13,765 DEMO.Row                                  0,0
 INFO 09:15:13,765 TESTBENCHTHREE.Row                        0,0
 WARN 09:15:13,765 Heap is 0.999394990192375 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
 INFO 09:15:13,765 Unable to reduce heap usage since there are no dirty
column families
 INFO 09:15:21,484 GC for ConcurrentMarkSweep: 7718 ms for 1 collections,
1568953672 used; max is 1569521664
 INFO 09:15:21,484 Pool Name                    Active   Pending   Blocked
 INFO 09:15:21,484 ReadStage                         0         0         0
 INFO 09:15:21,484 RequestResponseStage              0         0         0
 INFO 09:15:21,484 ReadRepairStage                   0         0         0
 INFO 09:15:21,484 MutationStage                     0         0         0
 INFO 09:15:21,484 ReplicateOnWriteStage             0         0         0
 INFO 09:15:21,484 GossipStage                       0         0         0
 INFO 09:15:21,484 AntiEntropyStage                  0         0         0
 INFO 09:15:21,500 MigrationStage                    0         0         0
 INFO 09:15:21,500 StreamStage                       0         0         0
 INFO 09:15:21,500 MemtablePostFlusher               0         0         0
 INFO 09:15:35,750 FlushWriter                       0         0         0
 INFO 09:15:35,750 MiscStage                         0         0         0
 INFO 09:15:35,750 commitlog_archiver                0         0         0
 INFO 09:15:35,750 InternalResponseStage             0         0         0
 INFO 09:15:35,750 HintedHandoff                     0         0         0
 INFO 09:15:35,750 CompactionManager                 0         0
 INFO 09:15:35,750 MessagingService                n/a       0,0
 INFO 09:15:35,750 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:16:26,046 KeyCache
8                        8
all
 INFO 09:16:31,140 RowCache
0                        0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:16:36,296 ColumnFamily                Memtable ops,data
 INFO 09:16:36,296 system.NodeIdInfo                         0,0
 INFO 09:16:36,296 system.IndexInfo                          0,0
 INFO 09:16:41,437 system.LocationInfo                       0,0
 INFO 09:16:41,437 system.Versions                           0,0
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid5316.hprof ...
Heap dump file created [1755520972 bytes in 84.007 secs]
 INFO 09:21:06,562 system.schema_keyspaces                   0,0
 INFO 09:24:11,906 Stop listening to thrift clients
 INFO 09:24:24,343 system.Migrations                         0,0
ERROR 09:24:24,343 Exception in thread Thread[COMMIT-LOG-WRITER,5,main]
java.lang.OutOfMemoryError: Java heap space
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1739)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1954)
    at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
    at
org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:46)
    at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    at java.lang.Thread.run(Thread.java:619)
ERROR 09:24:24,359 Exception in thread
Thread[PERIODIC-COMMIT-LOG-SYNCER,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR 09:24:24,359 Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR 09:24:24,359 Exception in thread Thread[RMI TCP
Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:725)
    at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
ERROR 09:24:24,359 Exception in thread Thread[Thrift:1,5,main]
java.lang.OutOfMemoryError: Java heap space
 INFO 09:24:24,343 system.schema_columnfamilies                 0,0
 INFO 09:24:24,421 system.schema_columns                     0,0
 INFO 09:24:24,421 system.HintsColumnFamily                  0,0
 INFO 09:24:24,421 system.Schema                             0,0
 INFO 09:24:24,421 Keyspace1.SuperCounter1                   0,0
 INFO 09:24:24,421 Keyspace1.Standard1                       0,0
 INFO 09:24:24,421 Keyspace1.Super1                          0,0
 INFO 09:24:24,421 Keyspace1.Counter1                        0,0
 INFO 09:24:24,421 TESTBENCHTWO.Row                          0,0
 INFO 09:24:24,421 TESTBENCH.Row                             0,0
 INFO 09:24:24,421 DEMO.Row                                  0,0
 INFO 09:24:24,421 TESTBENCHTHREE.Row                        0,0
 WARN 09:24:24,421 Heap is 0.9996381113985057 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
 INFO 09:24:24,421 Unable to reduce heap usage since there are no dirty
column families
 INFO 09:24:24,484 Announcing shutdown
 INFO 09:24:25,421 GC for ConcurrentMarkSweep: 463729 ms for 86
collections, 22709920 used; max is 1569521664
 INFO 09:24:25,421 Pool Name                    Active   Pending   Blocked
 INFO 09:24:25,421 ReadStage                         0         0         0
 INFO 09:24:25,421 RequestResponseStage              0         0         0
 INFO 09:24:25,421 ReadRepairStage                   0         0         0
 INFO 09:24:25,421 MutationStage                     0         0         0
 INFO 09:24:25,421 ReplicateOnWriteStage             0         0         0
 INFO 09:24:25,421 GossipStage                       0         0         0
 INFO 09:24:25,421 AntiEntropyStage                  0         0         0
 INFO 09:24:25,421 MigrationStage                    0         0         0
 INFO 09:24:25,437 StreamStage                       0         0         0
 INFO 09:24:25,437 MemtablePostFlusher               0         0         0
 INFO 09:24:25,437 FlushWriter                       0         0         0
 INFO 09:24:25,437 MiscStage                         0         0         0
 INFO 09:24:25,437 commitlog_archiver                0         0         0
 INFO 09:24:25,437 InternalResponseStage             0         0         0
 INFO 09:24:25,437 HintedHandoff                     0         0         0
 INFO 09:24:25,437 CompactionManager                 0         0
 INFO 09:24:25,437 MessagingService                n/a       0,0
 INFO 09:24:25,437 Cache Type                     Size
Capacity
KeysToSave                                                         Provider
 INFO 09:24:25,437 KeyCache
8                        8
all
 INFO 09:24:25,437 RowCache
0                        0                      all
org.apache.cassandra.cache.SerializingCacheProvider
 INFO 09:24:25,437 ColumnFamily                Memtable ops,data
 INFO 09:24:25,437 system.NodeIdInfo                         0,0
 INFO 09:24:25,437 system.IndexInfo                          0,0
 INFO 09:24:25,437 system.LocationInfo                       0,0
 INFO 09:24:25,437 system.Versions                           0,0
 INFO 09:24:25,437 system.schema_keyspaces                   0,0
 INFO 09:24:25,437 system.Migrations                         0,0
 INFO 09:24:25,437 system.schema_columnfamilies                 0,0
 INFO 09:24:25,437 system.schema_columns                     0,0
 INFO 09:24:25,437 system.HintsColumnFamily                  0,0
 INFO 09:24:25,437 system.Schema                             0,0
 INFO 09:24:25,437 Keyspace1.SuperCounter1                   0,0
 INFO 09:24:25,437 Keyspace1.Standard1                       0,0
 INFO 09:24:25,437 Keyspace1.Super1                          0,0
 INFO 09:24:25,437 Keyspace1.Counter1                        0,0
 INFO 09:24:25,437 TESTBENCHTWO.Row                          0,0
 INFO 09:24:25,437 TESTBENCH.Row                             0,0
 INFO 09:24:25,437 DEMO.Row                                  0,0
 INFO 09:24:25,437 TESTBENCHTHREE.Row                        0,0
 INFO 09:24:25,484 Waiting for messaging service to quiesce
 INFO 09:24:25,531 MessagingService shutting down server thread.

---- End of LOG -----

Mime
View raw message