cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Sterligov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8248) Possible memory leak
Date Thu, 06 Nov 2014 17:48:34 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200544#comment-14200544
] 

Alexander Sterligov commented on CASSANDRA-8248:
------------------------------------------------

No, it doesn't return to expected resources after forcing GC.

Just in case:

tpstats:
{quote}
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
CounterMutationStage              0         0              0         0                 0
ReadStage                         0         0        6868466         0                 0
RequestResponseStage              0         0      408799535         0                 0
MutationStage                     0         0      198570382         0                 0
ReadRepairStage                   0         0          46197         0                 0
GossipStage                       0         0        2150908         0                 0
CacheCleanupExecutor              0         0              0         0                 0
MigrationStage                    0         0            126         0                 0
ValidationExecutor                0         0          72747         0                 0
MemtableReclaimMemory             0         0          97987         0                 0
InternalResponseStage             0         0          41583         0                 0
AntiEntropyStage                  0         0         630829         0                 0
MiscStage                         0         0              0         0                 0
CommitLogArchiver                 0         0              0         0                 0
MemtableFlushWriter               0         0          97987         0                 0
PendingRangeCalculator            0         0             62         0                 0
MemtablePostFlush                 0         0         191444         0                 0
CompactionExecutor                0         0        4131622         0                 0
AntiEntropySessions               0         0            512         0                 0
HintedHandoff                     0         0            421         0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
PAGED_RANGE                  0
BINARY                       0
READ                         0
MUTATION                 30193
_TRACE                       0
REQUEST_RESPONSE             0
COUNTER_MUTATION             0
{quote}

compactionstats:
{quote}
pending tasks: 0
Active compaction remaining time :        n/a
{quote}

GC was just finished:
{quote}
INFO  [Service Thread] 2014-11-06 21:25:05,233 GCInspector.java:88 - ConcurrentMarkSweep GC
in 15275ms.  CMS Old Gen: 3596661592 -> 3596726792; Par Eden Space: 1371921168 -> 0;
{quote}

It still uses ~51g of RAM.


> Possible memory leak 
> ---------------------
>
>                 Key: CASSANDRA-8248
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alexander Sterligov
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports: 
> {quote}
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 15724 loadbase  20   0  493g  55g  44g S   28 44.2   4043:24 java
> {quote}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6332000-7f63a6562000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6562000-7f63a6792000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6792000-7f63a69c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6e22000-7f63a7052000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7052000-7f63a7282000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7282000-7f63a74b2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a74b2000-7f63a76e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a76e2000-7f63a7912000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7912000-7f63a7b42000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7b42000-7f63a7d72000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a81d2000-7f63a8402000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8402000-7f63a8622000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8622000-7f63a8842000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8842000-7f63a8a62000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8a62000-7f63a8c82000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8c82000-7f63a8ea2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8ea2000-7f63a90c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a90c2000-7f63a92e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> {quote}
> {quote}
> $ sudo grep deleted /proc/15724/maps | wc -l
> 640118
> {quote}
> {quote}
> $ sudo grep -v deleted /proc/15724/maps | wc -l
> 303340
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message