incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillermo Barbero <guillermo.barb...@spotbros.com>
Subject Re: too many warnings of Heap is full
Date Thu, 31 Jan 2013 12:34:19 GMT
Hi, replies to your answers:

The CFs that seems to flush more often (users and messages) do not
have multiget queries...

There are 4 secondary indexes for users and 3 for messages. I've seen
something odd here: the index of the CFs is also stored in a column
AND is also indexed. There isn't any query by that secondary index. I
should probably erase the index, shouldn't I?

I post some lines of the log before 21:35:

Node 150:
 INFO [HintedHandoff:1] 2013-01-30 21:07:43,795
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:07:43,796
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:17:43,796
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:17:43,796
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:27:43,796
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:27:43,797
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153

Node 151:
 INFO [HintedHandoff:1] 2013-01-30 21:04:45,821
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:04:45,822
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [ScheduledTasks:1] 2013-01-30 21:14:30,112 GCInspector.java
(line 122) GC for ParNew: 213 ms for 1 collections, 1920192480 used;
max is 3886022656
 INFO [HintedHandoff:1] 2013-01-30 21:14:45,821
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:14:45,822
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:24:45,822
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:24:45,822
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [ScheduledTasks:1] 2013-01-30 21:28:25,294 GCInspector.java
(line 122) GC for ParNew: 236 ms for 1 collections, 2264010544 used;
max is 3886022656
 INFO [HintedHandoff:1] 2013-01-30 21:34:45,822
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:34:45,822
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153

Node 152:
 INFO [HintedHandoff:1] 2013-01-30 21:02:03,706
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:02:03,707
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 WARN [MemoryMeter:1] 2013-01-30 21:09:57,220 Memtable.java (line 197)
setting live ratio to minimum of 1.0 instead of 0.15873949606408785
 INFO [MemoryMeter:1] 2013-01-30 21:09:57,221 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics') liveRatio is
1.0 (just-counted was 1.0).  calculation took 3ms for 301 columns
 INFO [MemoryMeter:1] 2013-01-30 21:09:57,223 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD',
ColumnFamily='CF_profilePics.CF_profilePics_pPicMD5_idx') liveRatio is
18.952013237727524 (just-counted was 18.952013237727524).  calculation
took 2ms for 74 columns
 INFO [MemoryMeter:1] 2013-01-30 21:09:57,242 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD',
ColumnFamily='CF_profilePics.CF_profilePics_userSBCode_idx') liveRatio
is 12.820966340897906 (just-counted was 12.209597352454495).
calculation took 19ms for 74 columns
 INFO [HintedHandoff:1] 2013-01-30 21:12:03,707
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:12:03,707
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [MemoryMeter:1] 2013-01-30 21:12:47,775 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD', ColumnFamily='CF_NewsFeedTimeline') liveRatio
is 3.605170679607766 (just-counted was 3.4883667978539283).
calculation took 110ms for 1462 columns
 INFO [HintedHandoff:1] 2013-01-30 21:22:03,707
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:22:03,708
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [MemoryMeter:1] 2013-01-30 21:24:14,795 Memtable.java (line 213)
CFS(Keyspace='SB_ACD', ColumnFamily='CF_GSMembers') liveRatio is
5.785411930027831 (just-counted was 5.622629145582293).  calculation
took 24ms for 463 columns
 WARN [ScheduledTasks:1] 2013-01-30 21:28:12,744 GCInspector.java
(line 145) Heap is 0.7609735541387436 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust
flush_largest_memtables_at threshold in cassandra.yaml if you don't
want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2013-01-30 21:28:12,744 StorageService.java
(line 2880) Flushing CFS(Keyspace='SB_ACD', ColumnFamily='CF_users')
to relieve memory pressure
 INFO [ScheduledTasks:1] 2013-01-30 21:28:12,745
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-CF_users.CF_users_userEmail_idx@1131307878(146/182
serialized/live bytes, 4 ops)
 INFO [ScheduledTasks:1] 2013-01-30 21:28:12,755
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-CF_users.CF_users_userOwnPhone_idx@973751772(146/182
serialized/live bytes, 4 ops)
 INFO [ScheduledTasks:1] 2013-01-30 21:28:12,755
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-CF_users.CF_users_userOwnPhoneKey_idx@2020641864(146/182
serialized/live bytes, 4 ops)
 INFO [ScheduledTasks:1] 2013-01-30 21:28:12,756
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-CF_users.CF_users_userSBCode_idx@621414590(146/182
serialized/live bytes, 4 ops)
 INFO [ScheduledTasks:1] 2013-01-30 21:28:12,756
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-CF_users@902038769(9358482/45802966 serialized/live bytes,
405140 ops)
 INFO [FlushWriter:11] 2013-01-30 21:28:12,757 Memtable.java (line
264) Writing Memtable-CF_users.CF_users_userEmail_idx@1131307878(146/182
serialized/live bytes, 4 ops)
 INFO [FlushWriter:11] 2013-01-30 21:28:12,768 Memtable.java (line
305) Completed flushing
/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userEmail_idx-hf-2231-Data.db
(234 bytes) for commitlog position
ReplayPosition(segmentId=1359573115280, position=6624766)
 INFO [FlushWriter:11] 2013-01-30 21:28:12,781 Memtable.java (line
264) Writing Memtable-CF_users.CF_users_userOwnPhone_idx@973751772(146/182
serialized/live bytes, 4 ops)
 INFO [FlushWriter:11] 2013-01-30 21:28:12,793 Memtable.java (line
305) Completed flushing
/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2217-Data.db
(265 bytes) for commitlog position
ReplayPosition(segmentId=1359573115280, position=6624766)
 INFO [FlushWriter:11] 2013-01-30 21:28:12,802 Memtable.java (line
264) Writing Memtable-CF_users.CF_users_userOwnPhoneKey_idx@2020641864(146/182
serialized/live bytes, 4 ops)
 INFO [CompactionExecutor:42] 2013-01-30 21:28:12,813
CompactionTask.java (line 109) Compacting
[SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2217-Data.db'),
SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2215-Data.db'),
SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2216-Data.db'),
SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2214-Data.db')]
 INFO [FlushWriter:11] 2013-01-30 21:28:12,814 Memtable.java (line
305) Completed flushing
/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2209-Data.db
(268 bytes) for commitlog position
ReplayPosition(segmentId=1359573115280, position=6624766)
 INFO [FlushWriter:11] 2013-01-30 21:28:12,824 Memtable.java (line
264) Writing Memtable-CF_users.CF_users_userSBCode_idx@621414590(146/182
serialized/live bytes, 4 ops)
 INFO [CompactionExecutor:43] 2013-01-30 21:28:12,826
CompactionTask.java (line 109) Compacting
[SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2206-Data.db'),
SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2208-Data.db'),
SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2209-Data.db'),
SSTableReader(path='/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2207-Data.db')]
 INFO [FlushWriter:11] 2013-01-30 21:28:12,835 Memtable.java (line
305) Completed flushing
/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userSBCode_idx-hf-2205-Data.db
(263 bytes) for commitlog position
ReplayPosition(segmentId=1359573115280, position=6624766)
 INFO [FlushWriter:11] 2013-01-30 21:28:12,838 Memtable.java (line
264) Writing Memtable-CF_users@902038769(9358482/45802966
serialized/live bytes, 405140 ops)
 INFO [FlushWriter:11] 2013-01-30 21:28:13,513 Memtable.java (line
305) Completed flushing
/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users-hf-3989-Data.db
(3358487 bytes) for commitlog position
ReplayPosition(segmentId=1359573115280, position=6624766)
 INFO [CompactionExecutor:42] 2013-01-30 21:28:28,209
CompactionTask.java (line 221) Compacted to
[/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhone_idx-hf-2218-Data.db,].
 12,758,536 to 12,758,266 (~99% of original) bytes for 298,733 keys at
0.790337MB/s.  Time: 15,395ms.
 INFO [CompactionExecutor:43] 2013-01-30 21:28:28,258
CompactionTask.java (line 221) Compacted to
[/raid0/cassandra/data/SB_ACD/CF_users/SB_ACD-CF_users.CF_users_userOwnPhoneKey_idx-hf-2210-Data.db,].
 12,827,751 to 12,827,331 (~99% of original) bytes for 298,767 keys at
0.792812MB/s.  Time: 15,430ms.
 INFO [MemoryMeter:1] 2013-01-30 21:31:30,638 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD', ColumnFamily='CF_userPhoneAgenda') liveRatio
is 9.966352958143455 (just-counted was 6.201877882350339).
calculation took 24ms for 6574 columns
 INFO [HintedHandoff:1] 2013-01-30 21:32:03,707
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:32:03,708
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 WARN [ScheduledTasks:1] 2013-01-30 21:33:41,079 GCInspector.java
(line 145) Heap is 0.7632044026868381 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust
flush_largest_memtables_at threshold in cassandra.yaml if you don't
want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2013-01-30 21:33:41,079 StorageService.java
(line 2880) Flushing CFS(Keyspace='SB_ACD',
ColumnFamily='CF_R2Notifications') to relieve memory pressure
 INFO [ScheduledTasks:1] 2013-01-30 21:33:41,080
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-CF_R2Notifications@1187465569(760532/33771575 serialized/live
bytes, 93136 ops)
 INFO [FlushWriter:12] 2013-01-30 21:33:41,105 Memtable.java (line
264) Writing Memtable-CF_R2Notifications@1187465569(760532/33771575
serialized/live bytes, 93136 ops)
 INFO [FlushWriter:12] 2013-01-30 21:33:42,802 Memtable.java (line
305) Completed flushing
/raid0/cassandra/data/SB_ACD/CF_R2Notifications/SB_ACD-CF_R2Notifications-hf-4175-Data.db
(639626 bytes) for commitlog position
ReplayPosition(segmentId=1359573115280, position=11032499)

Node 153:
 INFO [HintedHandoff:1] 2013-01-30 21:01:45,310
HintedHandOffManager.java (line 296) Started hinted handoff for token:
56713727820156407428984779325531226112 with IP: /10.0.0.152
 INFO [HintedHandoff:1] 2013-01-30 21:01:45,311
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.152
 INFO [MemoryMeter:1] 2013-01-30 21:05:45,297 Memtable.java (line 213)
CFS(Keyspace='SB_ACD', ColumnFamily='CF_users') liveRatio is
3.9183034832767603 (just-counted was 3.9183034832767603).  calculation
took 255ms for 58995 columns
 INFO [MemoryMeter:1] 2013-01-30 21:07:24,736 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD', ColumnFamily='CF_NewsFeedTimeline') liveRatio
is 3.6493489644964034 (just-counted was 3.471672980060775).
calculation took 107ms for 1395 columns
 INFO [ScheduledTasks:1] 2013-01-30 21:07:41,673 GCInspector.java
(line 122) GC for ParNew: 215 ms for 1 collections, 2310767440 used;
max is 3886022656
 INFO [HintedHandoff:1] 2013-01-30 21:11:45,316
HintedHandOffManager.java (line 296) Started hinted handoff for token:
56713727820156407428984779325531226112 with IP: /10.0.0.152
 INFO [HintedHandoff:1] 2013-01-30 21:11:45,318
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.152
 INFO [MemoryMeter:1] 2013-01-30 21:14:40,052 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD',
ColumnFamily='CF_profilePics.CF_profilePics_pPicMD5_idx') liveRatio is
18.87210642710637 (just-counted was 18.854822335025382).  calculation
took 3ms for 80 columns
 INFO [MemoryMeter:1] 2013-01-30 21:14:40,054 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD',
ColumnFamily='CF_profilePics.CF_profilePics_userSBCode_idx') liveRatio
is 12.252182911928061 (just-counted was 11.273365071410675).
calculation took 2ms for 81 columns
 WARN [MemoryMeter:1] 2013-01-30 21:14:40,072 Memtable.java (line 197)
setting live ratio to minimum of 1.0 instead of 0.12200431775304404
 INFO [MemoryMeter:1] 2013-01-30 21:14:40,072 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD', ColumnFamily='CF_profilePics') liveRatio is
1.0 (just-counted was 1.0).  calculation took 3ms for 315 columns
 INFO [ScheduledTasks:1] 2013-01-30 21:14:48,898 GCInspector.java
(line 122) GC for ParNew: 201 ms for 1 collections, 2512549552 used;
max is 3886022656
 INFO [MemoryMeter:1] 2013-01-30 21:15:32,061 Memtable.java (line 213)
CFS(Keyspace='SB_NNCD', ColumnFamily='CF_SBCodes') liveRatio is
32.58987080513604 (just-counted was 31.082180634662326).  calculation
took 2ms for 52 columns
 INFO [ScheduledTasks:1] 2013-01-30 21:21:08,322 GCInspector.java
(line 122) GC for ParNew: 239 ms for 1 collections, 2576069224 used;
max is 3886022656
 INFO [HintedHandoff:1] 2013-01-30 21:21:45,311
HintedHandOffManager.java (line 296) Started hinted handoff for token:
56713727820156407428984779325531226112 with IP: /10.0.0.152
 INFO [HintedHandoff:1] 2013-01-30 21:21:45,312
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.152
 INFO [MemoryMeter:1] 2013-01-30 21:23:46,635 Memtable.java (line 213)
CFS(Keyspace='SB_ACD', ColumnFamily='CF_GSData') liveRatio is
7.342488214798115 (just-counted was 7.342488214798115).  calculation
took 0ms for 116 columns
 INFO [MemoryMeter:1] 2013-01-30 21:24:14,793 Memtable.java (line 213)
CFS(Keyspace='SB_ACD', ColumnFamily='CF_GSMembers') liveRatio is
5.753869723542124 (just-counted was 5.630372102807786).  calculation
took 24ms for 456 columns
 INFO [ScheduledTasks:1] 2013-01-30 21:30:18,511 GCInspector.java
(line 122) GC for ParNew: 226 ms for 1 collections, 2242332984 used;
max is 3886022656
 INFO [MemoryMeter:1] 2013-01-30 21:30:46,426 Memtable.java (line 213)
CFS(Keyspace='SB_ACD', ColumnFamily='CF_R1Notifications') liveRatio is
6.832365653565714 (just-counted was 6.675185638944877).  calculation
took 43ms for 860 columns
 INFO [HintedHandoff:1] 2013-01-30 21:31:45,311
HintedHandOffManager.java (line 296) Started hinted handoff for token:
56713727820156407428984779325531226112 with IP: /10.0.0.152
 INFO [HintedHandoff:1] 2013-01-30 21:31:45,313
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.152
 INFO [ScheduledTasks:1] 2013-01-30 21:33:12,105 GCInspector.java
(line 122) GC for ParNew: 261 ms for 1 collections, 2603400728 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:33:21,312 GCInspector.java
(line 122) GC for ParNew: 248 ms for 1 collections, 3191815000 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:33:23,315 GCInspector.java
(line 122) GC for ParNew: 201 ms for 1 collections, 3471315456 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:33:35,323 GCInspector.java
(line 122) GC for ParNew: 204 ms for 1 collections, 1636739048 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:33:51,462 GCInspector.java
(line 122) GC for ParNew: 239 ms for 1 collections, 2789318016 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:00,805 GCInspector.java
(line 122) GC for ParNew: 217 ms for 1 collections, 2959924968 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:05,008 GCInspector.java
(line 122) GC for ParNew: 217 ms for 1 collections, 2900952792 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:17,262 GCInspector.java
(line 122) GC for ParNew: 231 ms for 1 collections, 2368279416 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:19,267 GCInspector.java
(line 122) GC for ParNew: 201 ms for 1 collections, 2297887816 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:28,326 GCInspector.java
(line 122) GC for ParNew: 215 ms for 1 collections, 2892129512 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:30,328 GCInspector.java
(line 122) GC for ParNew: 266 ms for 1 collections, 3168051328 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:42,552 GCInspector.java
(line 122) GC for ParNew: 249 ms for 1 collections, 2330298920 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:34:46,554 GCInspector.java
(line 122) GC for ParNew: 243 ms for 1 collections, 2496880592 used;
max is 3886022656
 WARN [ScheduledTasks:1] 2013-01-30 21:35:02,994 GCInspector.java
(line 145) Heap is 0.7641879728660028 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two
largest memtables to free up memory.  Adjust
flush_largest_memtables_at threshold in cassandra.yaml if you don't
want Cassandra to do this automatically
 WARN [ScheduledTasks:1] 2013-01-30 21:35:02,995 StorageService.java
(line 2880) Flushing CFS(Keyspace='SB_ACD',
ColumnFamily='CF_R2Notifications') to relieve memory pressure
 INFO [ScheduledTasks:1] 2013-01-30 21:35:02,995
ColumnFamilyStore.java (line 659) Enqueuing flush of
Memtable-CF_R2Notifications@1630759928(1137418/88800160
serialized/live bytes, 171909 ops)
 INFO [FlushWriter:4] 2013-01-30 21:35:03,016 Memtable.java (line 264)
Writing Memtable-CF_R2Notifications@1630759928(1137418/88800160
serialized/live bytes, 171909 ops)
 INFO [ScheduledTasks:1] 2013-01-30 21:35:04,293 GCInspector.java
(line 122) GC for ParNew: 455 ms for 2 collections, 2926515240 used;
max is 3886022656
 INFO [FlushWriter:4] 2013-01-30 21:35:05,798 Memtable.java (line 305)
Completed flushing
/raid0/cassandra/data/SB_ACD/CF_R2Notifications/SB_ACD-CF_R2Notifications-hf-3832-Data.db
(790256 bytes) for commitlog position
ReplayPosition(segmentId=1359573697867, position=11748063)
 INFO [ScheduledTasks:1] 2013-01-30 21:35:06,344 GCInspector.java
(line 122) GC for ParNew: 204 ms for 1 collections, 3227837984 used;
max is 3886022656
 INFO [MemoryMeter:1] 2013-01-30 21:35:06,369 Memtable.java (line 213)
CFS(Keyspace='SB_ACD', ColumnFamily='CF_R2Notifications') liveRatio is
37.62080440650366 (just-counted was 12.784246424797216).  calculation
took 7181ms for 2683 columns
 INFO [ScheduledTasks:1] 2013-01-30 21:35:08,345 GCInspector.java
(line 122) GC for ParNew: 216 ms for 1 collections, 3586092576 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:35:15,625 GCInspector.java
(line 122) GC for ConcurrentMarkSweep: 1130 ms for 2 collections,
602554696 used; max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:35:22,887 GCInspector.java
(line 122) GC for ParNew: 253 ms for 1 collections, 1362776544 used;
max is 3886022656
 INFO [ScheduledTasks:1] 2013-01-30 21:35:23,888 GCInspector.java
(line 122) GC for ParNew: 313 ms for 1 collections, 1473067952 used;
max is 3886022656

Node 154:
 INFO [HintedHandoff:1] 2013-01-30 21:05:00,728
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:05:00,728
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:15:00,728
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:15:00,729
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [ScheduledTasks:1] 2013-01-30 21:21:28,248 GCInspector.java
(line 122) GC for ParNew: 356 ms for 1 collections, 1272387016 used;
max is 3886022656
 INFO [HintedHandoff:1] 2013-01-30 21:25:00,728
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:25:00,729
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:35:00,728
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:35:00,729
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153

Node 155:
 INFO [HintedHandoff:1] 2013-01-30 21:05:07,916
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:05:07,917
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:15:07,916
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:15:07,917
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:25:07,916
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:25:07,918
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:35:07,917
HintedHandOffManager.java (line 296) Started hinted handoff for token:
85070591730234615865843651857942052864 with IP: /10.0.0.153
 INFO [HintedHandoff:1] 2013-01-30 21:35:07,918
HintedHandOffManager.java (line 392) Finished hinted handoff of 0 rows
to endpoint /10.0.0.153

I've read that the message 'Finished hinted handoff of 0 rows to
endpoint ...' is because a tombstone not being deleted, and that I
shouldn't worry about it.

I'm also talking with Amazon to check if there is a problem with their
service, the nodes 152 and 153 where using more cpu than the other
four nodes (but they have been like this all over the week, not just a
peak at the very same our for the last 3 days)

All the Cassandra configuration is at default values (the heap size,
row caches disabled, etc.)

Thanks for your support,


2013/1/31 Alain RODRIGUEZ <arodrime@gmail.com>:
> @Bryan
>
> "Other space is used but can be more easily controlled by tuning for
> - memtable
> - compaction
> - key cache
> - row cache"
>
> Isn't row cache stored off-heap since a while ?
>
>
> 2013/1/31 Bryan Talbot <btalbot@aeriagames.com>
>>
>> On Wed, Jan 30, 2013 at 2:44 PM, Guillermo Barbero
>> <guillermo.barbero@spotbros.com> wrote:
>>>
>>>  WARN [MemoryMeter:1] 2013-01-30 21:37:48,079 Memtable.java (line 202)
>>> setting live ratio to maximum of 64.0 instead of 751.6512549537648
>>>
>>
>> This looks interesting.  Doesn't this mean that the ratio of space used
>> for that CF for memory to serialized form is 751:1 but was forced to a lower
>> "sane" value?
>>
>> -Bryan
>>
>



-- 

Guillermo Barbero - Backend Team

Spotbros Technologies

Mime
View raw message