incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Talbot <btal...@aeriagames.com>
Subject Re: outOfMemory error
Date Wed, 28 Nov 2012 16:44:56 GMT
Well, asking for 500MB of data at once for a server with such modest
specs is asking for troubles.  Here are my suggestions.

Disable the 1 GB row cache
Consider allocating that memory for the java heap "Xms2500m Xmx2500m"
Don't fetch all the columns at once -- page through them a slice at a time
Increase the memtable to more than 64 MB if you want to write data to
this cluster

-Bryan



On Wed, Nov 28, 2012 at 5:06 AM, Damien Lejeune <d.lejeune@pepite.be> wrote:
> Hi all,
>
> I'm currently experiencing a outOfMemory problem with Cassandra-1.1.6 on
> Windows XP-Pro (32-bit). The server crashes when I try to query it with a
> relatively small amount of data (around 100 rows with 50000 columns each: to
> be precise, on my configuration, querying 75 or more rows makes the server
> to crash).
> I tried with different library (Hector, JDBC, Thrift) and with the Cassandra
> stress tool. All lead to the same outOfMemory problem.
>
> My dataset is composed, for each row, of: 1 column in DateType, 49999
> columns in DoubleType. I ran a query to fetch the entire dataset (around
> 330MB for the raw data + around 200MB for the metadata) and got the log at
> the end of this message.
>
> I also checked the heap-dump with Mat which displays these top values:
> Class Name
>              Objects                      Shallow Heap
> java.nio.HeapByteBuffer                       16,253,559
> 780,170,832
> bytes[]                                               16,254,013
> 330,207,640     <-- Data ?
> java.util.TreeMap$Entry                        8,126,711
> 260,054,752
> org.apache.cassandra.db.Column         8,116,589
> 194,798,136     <-- Metadata ?
>
> I tried to change the configuration in Cassandra for the values:
> - row_cache_size_in_mb: tried different value between [0,1000] MB
> - flush_largest_memtables_at: set to 0.1, but tried with 0.75
> - reduce_cache_sizes_at: tried 0.85, 0.6, 0.2 and 0.1
> - reduce_cache_capacity_to: tried 0.6 and 0.15
> - memtable_total_space_in_mb: 64 MB, but also tried to disable it (-> 1/3 of
> the heap)
> - Xms1G
> - Xmx1500M
> with no real observable improvements regarding my problem.
>
> My Cassandra server and client both run on the same machine.
>
> Here are the characteristics of my system configuration:
> - Cassandra-1.1.6
> - java version "1.6.0_20"
>          Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
>          Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)
> - Windows XP-Pro 32 bits with service pack 3
> - CPU double-core, 32 bits @2.26GHz
> - 3.48 of RAM
>
> I'm aware that my system configuration is not an optimized environment to
> make Cassandra to run efficiently, but I wonder if you guys know a
> workaround (or any idea on how) to fix this problem. Part of the answer is
> probably that I do not have enough RAM to run the process, but I also wonder
> if it is a 'normal' behaviour for Cassandra to handle this particular test
> case that way.
>
> Cheers,
>
> Damien
>
> ---- Cassandra's LOG ---
>
> Starting Cassandra Server
>  INFO 09:10:27,171 Logging initialized
>  INFO 09:10:27,171 JVM vendor/version: Java HotSpot(TM) Client VM/1.6.0_18
>  INFO 09:10:27,171 Heap size: 1072103424/1569521664
>  INFO 09:10:27,171 Classpath:
> E:\dl_benchmark\db\apache-cassandra-1.1.6\conf;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\antlr-3.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-clientutil-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-thrift-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\avro-1.4.0-fixes.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\avro-1.4.0-sources-fixes.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-cli-1.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-codec-1.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-lang-2.4.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\compress-lzf-0.8.4.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\concurrentlinkedhashmap-lru-1.3.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\guava-r08.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\high-scale-lib-1.1.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jackson-core-asl-1.9.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jackson-mapper-asl-1.9.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jamm-0.2.5.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jline-0.9.94.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\json-simple-1.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\libthrift-0.7.0.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\log4j-1.2.16.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\metrics-core-2.0.3.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\servlet-api-2.5-20081211.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\slf4j-api-1.6.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\slf4j-log4j12-1.6.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snakeyaml-1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snappy-java-1.0.4.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snaptree-0.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\build\classes\main;E:\dl_benchmark\db\apache-cassandra-1.1.6\build\classes\thrift;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jamm-0.2.5.jar
>  INFO 09:10:27,171 JNA not found. Native methods will be disabled.
>  INFO 09:10:27,187 Loading settings from
> file:/E:/dl_benchmark/db/apache-cassandra-1.1.6/conf/cassandra.yaml
>  INFO 09:10:27,296 32bit JVM detected.  It is recommended to run Cassandra
> on a 64bit JVM for better performance.
>  INFO 09:10:27,296 DiskAccessMode 'auto' determined to be standard,
> indexAccessMode is standard
>  INFO 09:10:27,500 Global memtable threshold is enabled at 64MB
>  INFO 09:10:27,781 Initializing key cache with capacity of 51 MBs.
>  INFO 09:10:27,812 Scheduling key cache save to each 14400 seconds (going to
> save all keys).
>  INFO 09:10:27,812 Initializing row cache with capacity of 1000 MBs and
> provider org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:10:27,812 Scheduling row cache save to each 0 seconds (going to
> save all keys).
>  INFO 09:10:27,906 reading saved cache
> \var\lib\cassandra\saved_caches\system-schema_columnfamilies-KeyCache
>  INFO 09:10:27,921 Opening
> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-hf-134
> (4938 bytes)
>  INFO 09:10:27,921 Opening
> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-hf-133
> (5122 bytes)
>  INFO 09:10:27,937 reading saved cache
> \var\lib\cassandra\saved_caches\system-schema_columns-KeyCache
>  INFO 09:10:27,937 Opening
> \var\lib\cassandra\data\system\schema_columns\system-schema_columns-hf-74
> (59 bytes)
>  INFO 09:10:27,937 Opening
> \var\lib\cassandra\data\system\schema_columns\system-schema_columns-hf-73
> (13836053 bytes)
>  INFO 09:10:27,953 Opening
> \var\lib\cassandra\data\system\Versions\system-Versions-hf-64 (247 bytes)
>  INFO 09:10:27,953 Opening
> \var\lib\cassandra\data\system\schema_keyspaces\system-schema_keyspaces-hf-105
> (1244 bytes)
>  INFO 09:10:27,953 reading saved cache
> \var\lib\cassandra\saved_caches\system-LocationInfo-KeyCache
>  INFO 09:10:27,953 Opening
> \var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-134 (346
> bytes)
>  INFO 09:10:32,843 reading saved cache
> \var\lib\cassandra\saved_caches\Keyspace1-Standard1-KeyCache
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-552
> (83124150 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-44
> (5541610 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-592
> (5541610 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-591
> (5541610 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-555
> (138540250 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-565
> (60957710 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-484
> (138540250 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-586
> (49874490 bytes)
>  INFO 09:10:32,859 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-437
> (138540250 bytes)
>  INFO 09:10:32,875 Opening
> \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-590
> (60957710 bytes)
>  INFO 09:10:32,875 Opening
> \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-133 (5514794
> bytes)
>  INFO 09:10:32,890 Opening
> \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-130 (195755351
> bytes)
>  INFO 09:10:32,906 Opening
> \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-131 (201295520
> bytes)
>  INFO 09:10:32,906 Opening
> \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-132 (30327963
> bytes)
>  INFO 09:10:32,906 Opening
> \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-129 (336374894
> bytes)
>  INFO 09:10:32,953 Opening
> \var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-198 (91008900 bytes)
>  INFO 09:10:32,968 Opening
> \var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-202 (57927906 bytes)
>  INFO 09:10:32,968 Opening
> \var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-188 (2917363977
> bytes)
>  INFO 09:10:33,156 Opening \var\lib\cassandra\data\DEMO\Row\DEMO-Row-hf-1
> (100 bytes)
>  INFO 09:10:33,171 Opening
> \var\lib\cassandra\data\TESTBENCHTHREE\Row\TESTBENCHTHREE-Row-hf-60
> (96519635 bytes)
>  INFO 09:10:33,171 Opening
> \var\lib\cassandra\data\TESTBENCHTHREE\Row\TESTBENCHTHREE-Row-hf-72
> (63929481 bytes)
>  INFO 09:10:43,890 completed pre-loading (10 keys) key cache.
>  INFO 09:10:44,000 Replaying
> \var\lib\cassandra\commitlog\CommitLog-1354090021204.log,
> \var\lib\cassandra\commitlog\CommitLog-1354090021205.log
>  INFO 09:10:44,015 Replaying
> \var\lib\cassandra\commitlog\CommitLog-1354090021204.log
>  INFO 09:10:44,031 Finished reading
> \var\lib\cassandra\commitlog\CommitLog-1354090021204.log
>  INFO 09:10:44,031 Replaying
> \var\lib\cassandra\commitlog\CommitLog-1354090021205.log
>  INFO 09:10:44,031 Finished reading
> \var\lib\cassandra\commitlog\CommitLog-1354090021205.log
>  INFO 09:10:44,046 Enqueuing flush of Memtable-Versions@6172767(83/103
> serialized/live bytes, 3 ops)
>  INFO 09:10:44,062 Writing Memtable-Versions@6172767(83/103 serialized/live
> bytes, 3 ops)
>  INFO 09:10:44,296 Completed flushing
> \var\lib\cassandra\data\system\Versions\system-Versions-hf-65-Data.db (247
> bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
> position=0)
>  INFO 09:10:44,312 Log replay complete, 3 replayed mutations
>  INFO 09:10:44,890 GC for ConcurrentMarkSweep: 210 ms for 1 collections,
> 41139480 used; max is 1569521664
>  INFO 09:10:49,906 Cassandra version: 1.1.6
>  INFO 09:10:49,906 Thrift API version: 19.32.0
>  INFO 09:10:49,906 CQL supported versions: 2.0.0,3.0.0-beta1 (default:
> 2.0.0)
>  INFO 09:10:49,953 Loading persisted ring state
>  INFO 09:10:49,953 Starting up server gossip
>  INFO 09:10:49,953 Enqueuing flush of Memtable-LocationInfo@8678192(29/36
> serialized/live bytes, 1 ops)
>  INFO 09:10:49,968 Writing Memtable-LocationInfo@8678192(29/36
> serialized/live bytes, 1 ops)
>  INFO 09:10:49,968 Completed flushing
> \var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-135-Data.db
> (80 bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
> position=363)
>  INFO 09:10:55,890 Starting Messaging Service on port 7000
>  INFO 09:10:55,890 Using saved token 6783448253301637730960311964445940653
>  INFO 09:10:55,906 Enqueuing flush of Memtable-LocationInfo@24013134(53/66
> serialized/live bytes, 2 ops)
>  INFO 09:10:55,906 Writing Memtable-LocationInfo@24013134(53/66
> serialized/live bytes, 2 ops)
>  INFO 09:10:55,906 Completed flushing
> \var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-136-Data.db
> (163 bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
> position=544)
>  INFO 09:10:55,906 Node localhost/127.0.0.1 state jump to normal
>  INFO 09:10:55,921 Bootstrap/Replace/Move completed! Now serving reads.
>  INFO 09:10:55,953 Binding thrift service to localhost/127.0.0.1:9160
>  INFO 09:10:55,953 Using TFastFramedTransport with a max frame size of
> 33554432 bytes.
>  INFO 09:10:55,953 Using synchronous/threadpool thrift server on
> localhost/127.0.0.1 : 9160
>  INFO 09:10:55,953 Listening for thrift clients...
>  --- The Querying starts here  ---
>  INFO 09:14:33,375 GC for ParNew: 742 ms for 1 collections, 1529127672 used;
> max is 1569521664
>  INFO 09:14:33,375 GC for ConcurrentMarkSweep: 7691 ms for 1 collections,
> 1529135040 used; max is 1569521664
>  INFO 09:14:33,421 Pool Name                    Active   Pending   Blocked
>  INFO 09:14:33,421 ReadStage                         0         0         0
>  INFO 09:14:33,421 RequestResponseStage              0         0         0
>  INFO 09:14:33,421 ReadRepairStage                   0         0         0
>  INFO 09:14:33,421 MutationStage                     0         0         0
>  INFO 09:14:33,421 ReplicateOnWriteStage             0         0         0
>  INFO 09:14:33,421 GossipStage                       0         0         0
>  INFO 09:14:33,421 AntiEntropyStage                  0         0         0
>  INFO 09:14:33,421 MigrationStage                    0         0         0
>  INFO 09:14:33,421 StreamStage                       0         0         0
>  INFO 09:14:33,421 MemtablePostFlusher               0         0         0
>  INFO 09:14:33,421 FlushWriter                       0         0         0
>  INFO 09:14:33,421 MiscStage                         0         0         0
>  INFO 09:14:33,421 commitlog_archiver                0         0         0
>  INFO 09:14:33,421 InternalResponseStage             0         0         0
>  INFO 09:14:33,421 HintedHandoff                     0         0         0
>  INFO 09:14:33,421 CompactionManager                 0         0
>  INFO 09:14:33,468 MessagingService                n/a       0,0
>  INFO 09:14:33,468 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:14:33,468 KeyCache                         14
> 1114112                      all
>  INFO 09:14:33,468 RowCache                          0
> 1048576000                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:14:33,468 ColumnFamily                Memtable ops,data
>  INFO 09:14:33,468 system.NodeIdInfo                         0,0
>  INFO 09:14:33,468 system.IndexInfo                          0,0
>  INFO 09:14:33,468 system.LocationInfo                       0,0
>  INFO 09:14:33,468 system.Versions                         3,103
>  INFO 09:14:33,468 system.schema_keyspaces                   0,0
>  INFO 09:14:33,468 system.Migrations                         0,0
>  INFO 09:14:33,468 system.schema_columnfamilies                 0,0
>  INFO 09:14:33,468 system.schema_columns                     0,0
>  INFO 09:14:33,468 system.HintsColumnFamily                  0,0
>  INFO 09:14:33,468 system.Schema                             0,0
>  INFO 09:14:33,468 Keyspace1.SuperCounter1                   0,0
>  INFO 09:14:33,468 Keyspace1.Standard1                       0,0
>  INFO 09:14:33,468 Keyspace1.Super1                          0,0
>  INFO 09:14:33,468 Keyspace1.Counter1                        0,0
>  INFO 09:14:33,468 TESTBENCHTWO.Row                          0,0
>  INFO 09:14:33,468 TESTBENCH.Row                             0,0
>  INFO 09:14:33,468 DEMO.Row                                  0,0
>  INFO 09:14:33,468 TESTBENCHTHREE.Row                        0,0
>  WARN 09:14:33,468 Heap is 0.9742681958928348 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra is now reducing cache sizes to free
> up memory.  Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you
> don't want Cassandra to do this automatically
>  WARN 09:14:33,468 Reducing RowCache capacity from 1048576000 to 0 to reduce
> memory pressure
>  WARN 09:14:33,468 Reducing KeyCache capacity from 1114112 to 8 to reduce
> memory pressure
>  WARN 09:14:33,468 Heap is 0.9742681958928348 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  WARN 09:14:33,468 Flushing CFS(Keyspace='system', ColumnFamily='Versions')
> to relieve memory pressure
>  INFO 09:14:33,468 Enqueuing flush of Memtable-Versions@9422607(83/103
> serialized/live bytes, 3 ops)
>  INFO 09:14:33,468 Writing Memtable-Versions@9422607(83/103 serialized/live
> bytes, 3 ops)
>  INFO 09:14:33,562 Completed flushing
> \var\lib\cassandra\data\system\Versions\system-Versions-hf-66-Data.db (247
> bytes) for commitlog position ReplayPosition(segmentId=1354090244001,
> position=544)
>  INFO 09:14:42,968 GC for ConcurrentMarkSweep: 8900 ms for 1 collections,
> 1549381648 used; max is 1569521664
>  INFO 09:14:42,968 Pool Name                    Active   Pending   Blocked
>  INFO 09:14:42,968 ReadStage                         0         0         0
>  INFO 09:14:42,968 RequestResponseStage              0         0         0
>  INFO 09:14:42,984 ReadRepairStage                   0         0         0
>  INFO 09:14:42,984 MutationStage                     0         0         0
>  INFO 09:14:42,984 ReplicateOnWriteStage             0         0         0
>  INFO 09:14:43,015 GossipStage                       0         0         0
>  INFO 09:14:43,015 AntiEntropyStage                  0         0         0
>  INFO 09:14:43,015 MigrationStage                    0         0         0
>  INFO 09:14:43,015 StreamStage                       0         0         0
>  INFO 09:14:43,015 MemtablePostFlusher               0         0         0
>  INFO 09:14:43,015 FlushWriter                       0         0         0
>  INFO 09:14:43,015 MiscStage                         0         0         0
>  INFO 09:14:43,015 commitlog_archiver                0         0         0
>  INFO 09:14:43,015 InternalResponseStage             0         0         0
>  INFO 09:14:43,015 HintedHandoff                     0         0         0
>  INFO 09:14:43,015 CompactionManager                 0         0
>  INFO 09:14:43,015 MessagingService                n/a       0,0
>  INFO 09:14:43,015 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:14:43,015 KeyCache                          8
> 8                      all
>  INFO 09:14:43,015 RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:14:43,015 ColumnFamily                Memtable ops,data
>  INFO 09:14:43,015 system.NodeIdInfo                         0,0
>  INFO 09:14:43,015 system.IndexInfo                          0,0
>  INFO 09:14:43,015 system.LocationInfo                       0,0
>  INFO 09:14:43,015 system.Versions                           0,0
>  INFO 09:14:43,015 system.schema_keyspaces                   0,0
>  INFO 09:14:43,015 system.Migrations                         0,0
>  INFO 09:14:43,015 system.schema_columnfamilies                 0,0
>  INFO 09:14:43,015 system.schema_columns                     0,0
>  INFO 09:14:43,015 system.HintsColumnFamily                  0,0
>  INFO 09:14:43,015 system.Schema                             0,0
>  INFO 09:14:43,015 Keyspace1.SuperCounter1                   0,0
>  INFO 09:14:43,015 Keyspace1.Standard1                       0,0
>  INFO 09:14:43,015 Keyspace1.Super1                          0,0
>  INFO 09:14:43,015 Keyspace1.Counter1                        0,0
>  INFO 09:14:43,015 TESTBENCHTWO.Row                          0,0
>  INFO 09:14:43,015 TESTBENCH.Row                             0,0
>  INFO 09:14:43,015 DEMO.Row                                  0,0
>  INFO 09:14:43,015 TESTBENCHTHREE.Row                        0,0
>  WARN 09:14:43,015 Heap is 0.9871680547889525 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO 09:14:43,015 Unable to reduce heap usage since there are no dirty
> column families
>  INFO 09:14:50,046 GC for ConcurrentMarkSweep: 6530 ms for 1 collections,
> 1561411240 used; max is 1569521664
>  INFO 09:14:50,234 Pool Name                    Active   Pending   Blocked
>  INFO 09:14:50,234 ReadStage                         0         0         0
>  INFO 09:14:50,234 RequestResponseStage              0         0         0
>  INFO 09:14:50,234 ReadRepairStage                   0         0         0
>  INFO 09:14:50,234 MutationStage                     0         0         0
>  INFO 09:14:50,234 ReplicateOnWriteStage             0         0         0
>  INFO 09:14:50,234 GossipStage                       0         0         0
>  INFO 09:14:50,234 AntiEntropyStage                  0         0         0
>  INFO 09:14:50,234 MigrationStage                    0         0         0
>  INFO 09:14:50,234 StreamStage                       0         0         0
>  INFO 09:14:50,234 MemtablePostFlusher               0         0         0
>  INFO 09:14:50,234 FlushWriter                       0         0         0
>  INFO 09:14:50,234 MiscStage                         0         0         0
>  INFO 09:14:50,234 commitlog_archiver                0         0         0
>  INFO 09:14:50,234 InternalResponseStage             0         0         0
>  INFO 09:14:50,234 HintedHandoff                     0         0         0
>  INFO 09:14:50,234 CompactionManager                 0         0
>  INFO 09:14:50,234 MessagingService                n/a       0,0
>  INFO 09:14:50,234 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:14:50,234 KeyCache                          8
> 8                      all
>  INFO 09:14:50,234 RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:14:50,234 ColumnFamily                Memtable ops,data
>  INFO 09:14:50,234 system.NodeIdInfo                         0,0
>  INFO 09:14:50,234 system.IndexInfo                          0,0
>  INFO 09:14:50,234 system.LocationInfo                       0,0
>  INFO 09:14:50,234 system.Versions                           0,0
>  INFO 09:14:50,234 system.schema_keyspaces                   0,0
>  INFO 09:14:50,234 system.Migrations                         0,0
>  INFO 09:14:50,234 system.schema_columnfamilies                 0,0
>  INFO 09:14:50,234 system.schema_columns                     0,0
>  INFO 09:14:50,265 system.HintsColumnFamily                  0,0
>  INFO 09:14:50,265 system.Schema                             0,0
>  INFO 09:14:50,265 Keyspace1.SuperCounter1                   0,0
>  INFO 09:14:50,265 Keyspace1.Standard1                       0,0
>  INFO 09:14:50,265 Keyspace1.Super1                          0,0
>  INFO 09:14:50,265 Keyspace1.Counter1                        0,0
>  INFO 09:14:50,265 TESTBENCHTWO.Row                          0,0
>  INFO 09:14:50,265 TESTBENCH.Row                             0,0
>  INFO 09:14:50,265 DEMO.Row                                  0,0
>  INFO 09:14:50,265 TESTBENCHTHREE.Row                        0,0
>  WARN 09:14:50,265 Heap is 0.9948325504604185 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO 09:14:50,265 Unable to reduce heap usage since there are no dirty
> column families
>  INFO 09:14:59,343 GC for ConcurrentMarkSweep: 8724 ms for 1 collections,
> 1566184304 used; max is 1569521664
>  INFO 09:14:59,343 Pool Name                    Active   Pending   Blocked
>  INFO 09:14:59,343 ReadStage                         0         0         0
>  INFO 09:14:59,343 RequestResponseStage              0         0         0
>  INFO 09:14:59,343 ReadRepairStage                   0         0         0
>  INFO 09:14:59,343 MutationStage                     0         0         0
>  INFO 09:14:59,343 ReplicateOnWriteStage             0         0         0
>  INFO 09:14:59,343 GossipStage                       0         0         0
>  INFO 09:14:59,343 AntiEntropyStage                  0         0         0
>  INFO 09:14:59,343 MigrationStage                    0         0         0
>  INFO 09:14:59,343 StreamStage                       0         0         0
>  INFO 09:14:59,343 MemtablePostFlusher               0         0         0
>  INFO 09:14:59,343 FlushWriter                       0         0         0
>  INFO 09:14:59,343 MiscStage                         0         0         0
>  INFO 09:14:59,343 commitlog_archiver                0         0         0
>  INFO 09:14:59,343 InternalResponseStage             0         0         0
>  INFO 09:14:59,343 HintedHandoff                     0         0         0
>  INFO 09:14:59,343 CompactionManager                 0         0
>  INFO 09:14:59,343 MessagingService                n/a       0,0
>  INFO 09:14:59,343 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:14:59,343 KeyCache                          8
> 8                      all
>  INFO 09:14:59,343 RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:14:59,343 ColumnFamily                Memtable ops,data
>  INFO 09:14:59,343 system.NodeIdInfo                         0,0
>  INFO 09:14:59,343 system.IndexInfo                          0,0
>  INFO 09:14:59,343 system.LocationInfo                       0,0
>  INFO 09:14:59,343 system.Versions                           0,0
>  INFO 09:14:59,343 system.schema_keyspaces                   0,0
>  INFO 09:14:59,343 system.Migrations                         0,0
>  INFO 09:14:59,343 system.schema_columnfamilies                 0,0
>  INFO 09:14:59,343 system.schema_columns                     0,0
>  INFO 09:14:59,343 system.HintsColumnFamily                  0,0
>  INFO 09:14:59,343 system.Schema                             0,0
>  INFO 09:14:59,343 Keyspace1.SuperCounter1                   0,0
>  INFO 09:14:59,343 Keyspace1.Standard1                       0,0
>  INFO 09:14:59,343 Keyspace1.Super1                          0,0
>  INFO 09:14:59,343 Keyspace1.Counter1                        0,0
>  INFO 09:14:59,343 TESTBENCHTWO.Row                          0,0
>  INFO 09:14:59,343 TESTBENCH.Row                             0,0
>  INFO 09:14:59,343 DEMO.Row                                  0,0
>  INFO 09:14:59,343 TESTBENCHTHREE.Row                        0,0
>  WARN 09:14:59,343 Heap is 0.997873645151546 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO 09:14:59,343 Unable to reduce heap usage since there are no dirty
> column families
>  INFO 09:15:08,500 GC for ConcurrentMarkSweep: 8854 ms for 1 collections,
> 1567778080 used; max is 1569521664
>  INFO 09:15:08,500 Pool Name                    Active   Pending   Blocked
>  INFO 09:15:08,500 ReadStage                         0         0         0
>  INFO 09:15:08,500 RequestResponseStage              0         0         0
>  INFO 09:15:08,500 ReadRepairStage                   0         0         0
>  INFO 09:15:08,500 MutationStage                     0         0         0
>  INFO 09:15:08,500 ReplicateOnWriteStage             0         0         0
>  INFO 09:15:08,500 GossipStage                       0         0         0
>  INFO 09:15:08,515 AntiEntropyStage                  0         0         0
>  INFO 09:15:08,515 MigrationStage                    0         0         0
>  INFO 09:15:08,515 StreamStage                       0         0         0
>  INFO 09:15:08,515 MemtablePostFlusher               0         0         0
>  INFO 09:15:08,515 FlushWriter                       0         0         0
>  INFO 09:15:08,515 MiscStage                         0         0         0
>  INFO 09:15:08,515 commitlog_archiver                0         0         0
>  INFO 09:15:08,515 InternalResponseStage             0         0         0
>  INFO 09:15:08,515 HintedHandoff                     0         0         0
>  INFO 09:15:08,515 CompactionManager                 0         0
>  INFO 09:15:08,515 MessagingService                n/a       0,0
>  INFO 09:15:08,515 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:15:08,515 KeyCache                          8
> 8                      all
>  INFO 09:15:08,515 RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:15:08,515 ColumnFamily                Memtable ops,data
>  INFO 09:15:08,515 system.NodeIdInfo                         0,0
>  INFO 09:15:08,515 system.IndexInfo                          0,0
>  INFO 09:15:08,515 system.LocationInfo                       0,0
>  INFO 09:15:08,515 system.Versions                           0,0
>  INFO 09:15:08,515 system.schema_keyspaces                   0,0
>  INFO 09:15:08,515 system.Migrations                         0,0
>  INFO 09:15:08,515 system.schema_columnfamilies                 0,0
>  INFO 09:15:08,515 system.schema_columns                     0,0
>  INFO 09:15:08,515 system.HintsColumnFamily                  0,0
>  INFO 09:15:08,515 system.Schema                             0,0
>  INFO 09:15:08,515 Keyspace1.SuperCounter1                   0,0
>  INFO 09:15:08,515 Keyspace1.Standard1                       0,0
>  INFO 09:15:08,515 Keyspace1.Super1                          0,0
>  INFO 09:15:08,515 Keyspace1.Counter1                        0,0
>  INFO 09:15:08,515 TESTBENCHTWO.Row                          0,0
>  INFO 09:15:08,515 TESTBENCH.Row                             0,0
>  INFO 09:15:08,515 DEMO.Row                                  0,0
>  INFO 09:15:08,515 TESTBENCHTHREE.Row                        0,0
>  WARN 09:15:08,515 Heap is 0.9988890984814084 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO 09:15:08,515 Unable to reduce heap usage since there are no dirty
> column families
>  INFO 09:15:13,734 GC for ConcurrentMarkSweep: 5129 ms for 1 collections,
> 1568572088 used; max is 1569521664
>  INFO 09:15:13,734 Pool Name                    Active   Pending   Blocked
>  INFO 09:15:13,734 ReadStage                         0         0         0
>  INFO 09:15:13,734 RequestResponseStage              0         0         0
>  INFO 09:15:13,734 ReadRepairStage                   0         0         0
>  INFO 09:15:13,734 MutationStage                     0         0         0
>  INFO 09:15:13,734 ReplicateOnWriteStage             0         0         0
>  INFO 09:15:13,734 GossipStage                       0         0         0
>  INFO 09:15:13,734 AntiEntropyStage                  0         0         0
>  INFO 09:15:13,734 MigrationStage                    0         0         0
>  INFO 09:15:13,734 StreamStage                       0         0         0
>  INFO 09:15:13,734 MemtablePostFlusher               0         0         0
>  INFO 09:15:13,734 FlushWriter                       0         0         0
>  INFO 09:15:13,734 MiscStage                         0         0         0
>  INFO 09:15:13,765 commitlog_archiver                0         0         0
>  INFO 09:15:13,765 InternalResponseStage             0         0         0
>  INFO 09:15:13,765 HintedHandoff                     0         0         0
>  INFO 09:15:13,765 CompactionManager                 0         0
>  INFO 09:15:13,765 MessagingService                n/a       0,0
>  INFO 09:15:13,765 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:15:13,765 KeyCache                          8
> 8                      all
>  INFO 09:15:13,765 RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:15:13,765 ColumnFamily                Memtable ops,data
>  INFO 09:15:13,765 system.NodeIdInfo                         0,0
>  INFO 09:15:13,765 system.IndexInfo                          0,0
>  INFO 09:15:13,765 system.LocationInfo                       0,0
>  INFO 09:15:13,765 system.Versions                           0,0
>  INFO 09:15:13,765 system.schema_keyspaces                   0,0
>  INFO 09:15:13,765 system.Migrations                         0,0
>  INFO 09:15:13,765 system.schema_columnfamilies                 0,0
>  INFO 09:15:13,765 system.schema_columns                     0,0
>  INFO 09:15:13,765 system.HintsColumnFamily                  0,0
>  INFO 09:15:13,765 system.Schema                             0,0
>  INFO 09:15:13,765 Keyspace1.SuperCounter1                   0,0
>  INFO 09:15:13,765 Keyspace1.Standard1                       0,0
>  INFO 09:15:13,765 Keyspace1.Super1                          0,0
>  INFO 09:15:13,765 Keyspace1.Counter1                        0,0
>  INFO 09:15:13,765 TESTBENCHTWO.Row                          0,0
>  INFO 09:15:13,765 TESTBENCH.Row                             0,0
>  INFO 09:15:13,765 DEMO.Row                                  0,0
>  INFO 09:15:13,765 TESTBENCHTHREE.Row                        0,0
>  WARN 09:15:13,765 Heap is 0.999394990192375 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO 09:15:13,765 Unable to reduce heap usage since there are no dirty
> column families
>  INFO 09:15:21,484 GC for ConcurrentMarkSweep: 7718 ms for 1 collections,
> 1568953672 used; max is 1569521664
>  INFO 09:15:21,484 Pool Name                    Active   Pending   Blocked
>  INFO 09:15:21,484 ReadStage                         0         0         0
>  INFO 09:15:21,484 RequestResponseStage              0         0         0
>  INFO 09:15:21,484 ReadRepairStage                   0         0         0
>  INFO 09:15:21,484 MutationStage                     0         0         0
>  INFO 09:15:21,484 ReplicateOnWriteStage             0         0         0
>  INFO 09:15:21,484 GossipStage                       0         0         0
>  INFO 09:15:21,484 AntiEntropyStage                  0         0         0
>  INFO 09:15:21,500 MigrationStage                    0         0         0
>  INFO 09:15:21,500 StreamStage                       0         0         0
>  INFO 09:15:21,500 MemtablePostFlusher               0         0         0
>  INFO 09:15:35,750 FlushWriter                       0         0         0
>  INFO 09:15:35,750 MiscStage                         0         0         0
>  INFO 09:15:35,750 commitlog_archiver                0         0         0
>  INFO 09:15:35,750 InternalResponseStage             0         0         0
>  INFO 09:15:35,750 HintedHandoff                     0         0         0
>  INFO 09:15:35,750 CompactionManager                 0         0
>  INFO 09:15:35,750 MessagingService                n/a       0,0
>  INFO 09:15:35,750 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:16:26,046 KeyCache                          8
> 8                      all
>  INFO 09:16:31,140 RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:16:36,296 ColumnFamily                Memtable ops,data
>  INFO 09:16:36,296 system.NodeIdInfo                         0,0
>  INFO 09:16:36,296 system.IndexInfo                          0,0
>  INFO 09:16:41,437 system.LocationInfo                       0,0
>  INFO 09:16:41,437 system.Versions                           0,0
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid5316.hprof ...
> Heap dump file created [1755520972 bytes in 84.007 secs]
>  INFO 09:21:06,562 system.schema_keyspaces                   0,0
>  INFO 09:24:11,906 Stop listening to thrift clients
>  INFO 09:24:24,343 system.Migrations                         0,0
> ERROR 09:24:24,343 Exception in thread Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.OutOfMemoryError: Java heap space
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1739)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1954)
>     at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>     at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:46)
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>     at java.lang.Thread.run(Thread.java:619)
> ERROR 09:24:24,359 Exception in thread
> Thread[PERIODIC-COMMIT-LOG-SYNCER,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR 09:24:24,359 Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR 09:24:24,359 Exception in thread Thread[RMI TCP Connection(idle),5,RMI
> Runtime]
> java.lang.OutOfMemoryError: Java heap space
>     at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
>     at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:725)
>     at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:619)
> ERROR 09:24:24,359 Exception in thread Thread[Thrift:1,5,main]
> java.lang.OutOfMemoryError: Java heap space
>  INFO 09:24:24,343 system.schema_columnfamilies                 0,0
>  INFO 09:24:24,421 system.schema_columns                     0,0
>  INFO 09:24:24,421 system.HintsColumnFamily                  0,0
>  INFO 09:24:24,421 system.Schema                             0,0
>  INFO 09:24:24,421 Keyspace1.SuperCounter1                   0,0
>  INFO 09:24:24,421 Keyspace1.Standard1                       0,0
>  INFO 09:24:24,421 Keyspace1.Super1                          0,0
>  INFO 09:24:24,421 Keyspace1.Counter1                        0,0
>  INFO 09:24:24,421 TESTBENCHTWO.Row                          0,0
>  INFO 09:24:24,421 TESTBENCH.Row                             0,0
>  INFO 09:24:24,421 DEMO.Row                                  0,0
>  INFO 09:24:24,421 TESTBENCHTHREE.Row                        0,0
>  WARN 09:24:24,421 Heap is 0.9996381113985057 full.  You may need to reduce
> memtable and/or cache sizes.  Cassandra will now flush up to the two largest
> memtables to free up memory.  Adjust flush_largest_memtables_at threshold in
> cassandra.yaml if you don't want Cassandra to do this automatically
>  INFO 09:24:24,421 Unable to reduce heap usage since there are no dirty
> column families
>  INFO 09:24:24,484 Announcing shutdown
>  INFO 09:24:25,421 GC for ConcurrentMarkSweep: 463729 ms for 86 collections,
> 22709920 used; max is 1569521664
>  INFO 09:24:25,421 Pool Name                    Active   Pending   Blocked
>  INFO 09:24:25,421 ReadStage                         0         0         0
>  INFO 09:24:25,421 RequestResponseStage              0         0         0
>  INFO 09:24:25,421 ReadRepairStage                   0         0         0
>  INFO 09:24:25,421 MutationStage                     0         0         0
>  INFO 09:24:25,421 ReplicateOnWriteStage             0         0         0
>  INFO 09:24:25,421 GossipStage                       0         0         0
>  INFO 09:24:25,421 AntiEntropyStage                  0         0         0
>  INFO 09:24:25,421 MigrationStage                    0         0         0
>  INFO 09:24:25,437 StreamStage                       0         0         0
>  INFO 09:24:25,437 MemtablePostFlusher               0         0         0
>  INFO 09:24:25,437 FlushWriter                       0         0         0
>  INFO 09:24:25,437 MiscStage                         0         0         0
>  INFO 09:24:25,437 commitlog_archiver                0         0         0
>  INFO 09:24:25,437 InternalResponseStage             0         0         0
>  INFO 09:24:25,437 HintedHandoff                     0         0         0
>  INFO 09:24:25,437 CompactionManager                 0         0
>  INFO 09:24:25,437 MessagingService                n/a       0,0
>  INFO 09:24:25,437 Cache Type                     Size
> Capacity               KeysToSave
> Provider
>  INFO 09:24:25,437 KeyCache                          8
> 8                      all
>  INFO 09:24:25,437 RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
>  INFO 09:24:25,437 ColumnFamily                Memtable ops,data
>  INFO 09:24:25,437 system.NodeIdInfo                         0,0
>  INFO 09:24:25,437 system.IndexInfo                          0,0
>  INFO 09:24:25,437 system.LocationInfo                       0,0
>  INFO 09:24:25,437 system.Versions                           0,0
>  INFO 09:24:25,437 system.schema_keyspaces                   0,0
>  INFO 09:24:25,437 system.Migrations                         0,0
>  INFO 09:24:25,437 system.schema_columnfamilies                 0,0
>  INFO 09:24:25,437 system.schema_columns                     0,0
>  INFO 09:24:25,437 system.HintsColumnFamily                  0,0
>  INFO 09:24:25,437 system.Schema                             0,0
>  INFO 09:24:25,437 Keyspace1.SuperCounter1                   0,0
>  INFO 09:24:25,437 Keyspace1.Standard1                       0,0
>  INFO 09:24:25,437 Keyspace1.Super1                          0,0
>  INFO 09:24:25,437 Keyspace1.Counter1                        0,0
>  INFO 09:24:25,437 TESTBENCHTWO.Row                          0,0
>  INFO 09:24:25,437 TESTBENCH.Row                             0,0
>  INFO 09:24:25,437 DEMO.Row                                  0,0
>  INFO 09:24:25,437 TESTBENCHTHREE.Row                        0,0
>  INFO 09:24:25,484 Waiting for messaging service to quiesce
>  INFO 09:24:25,531 MessagingService shutting down server thread.
>
> ---- End of LOG -----

Mime
View raw message