cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Sterligov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-8248) Possible memory leak
Date Tue, 04 Nov 2014 00:20:34 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alexander Sterligov updated CASSANDRA-8248:
-------------------------------------------
    Description: 
Sometimes during repair cassandra starts to consume more memory than expected.

Total amount of data on node is about 20GB.
Size of the data directory is 66GC because of snapshots.

Top reports: 
{quote}
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15724 loadbase  20   0  493g  55g  44g S   28 44.2   4043:24 java
{quote}

At the /proc/15724/maps there are a lot of deleted file maps
{quote}
7f63a6102000-7f63a6332000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6332000-7f63a6562000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6562000-7f63a6792000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6792000-7f63a69c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a69c2000-7f63a6bf2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6bf2000-7f63a6e22000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6e22000-7f63a7052000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7052000-7f63a7282000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7282000-7f63a74b2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a74b2000-7f63a76e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a76e2000-7f63a7912000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7912000-7f63a7b42000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7b42000-7f63a7d72000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7d72000-7f63a7fa2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7fa2000-7f63a81d2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a81d2000-7f63a8402000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8402000-7f63a8622000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8622000-7f63a8842000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8842000-7f63a8a62000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8a62000-7f63a8c82000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8c82000-7f63a8ea2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8ea2000-7f63a90c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a90c2000-7f63a92e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
{quote}

{quote}
$ sudo grep deleted /proc/15724/maps | wc -l
640118
{quote}

{quote}
$ sudo grep -v deleted /proc/15724/maps | wc -l
303340
{quote}

  was:
Sometimes during repair cassandra starts to consume more memory than expected.

Total amount of data on node is about 20GB.
Size of the data directory is 66GC because of snapshots.

Top reports: 
{quote}
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15724 loadbase  20   0  493g  55g  44g S   28 44.2   4043:24 java
{quote}

At the /proc/15724/maps there are a lot of deleted file maps
{quote}
7f63a6102000-7f63a6332000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6332000-7f63a6562000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6562000-7f63a6792000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6792000-7f63a69c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a69c2000-7f63a6bf2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6bf2000-7f63a6e22000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a6e22000-7f63a7052000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7052000-7f63a7282000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7282000-7f63a74b2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a74b2000-7f63a76e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a76e2000-7f63a7912000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7912000-7f63a7b42000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7b42000-7f63a7d72000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7d72000-7f63a7fa2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a7fa2000-7f63a81d2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a81d2000-7f63a8402000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8402000-7f63a8622000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8622000-7f63a8842000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8842000-7f63a8a62000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8a62000-7f63a8c82000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8c82000-7f63a8ea2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a8ea2000-7f63a90c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
7f63a90c2000-7f63a92e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
{quote}

{quote}
$ sudo grep -v deleted /proc/15724/maps | wc -l
303340
{quote}

{quote}
$ sudo grep deleted /proc/15724/maps | wc -l
640118
{quote}

{quote}
$ sudo grep -v deleted /proc/15724/maps | wc -l
303340
{quote}


> Possible memory leak 
> ---------------------
>
>                 Key: CASSANDRA-8248
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8248
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alexander Sterligov
>
> Sometimes during repair cassandra starts to consume more memory than expected.
> Total amount of data on node is about 20GB.
> Size of the data directory is 66GC because of snapshots.
> Top reports: 
> {quote}
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 15724 loadbase  20   0  493g  55g  44g S   28 44.2   4043:24 java
> {quote}
> At the /proc/15724/maps there are a lot of deleted file maps
> {quote}
> 7f63a6102000-7f63a6332000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6332000-7f63a6562000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6562000-7f63a6792000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6792000-7f63a69c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a69c2000-7f63a6bf2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6bf2000-7f63a6e22000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a6e22000-7f63a7052000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7052000-7f63a7282000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7282000-7f63a74b2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a74b2000-7f63a76e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a76e2000-7f63a7912000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7912000-7f63a7b42000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7b42000-7f63a7d72000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7d72000-7f63a7fa2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a7fa2000-7f63a81d2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a81d2000-7f63a8402000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8402000-7f63a8622000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8622000-7f63a8842000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8842000-7f63a8a62000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8a62000-7f63a8c82000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8c82000-7f63a8ea2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a8ea2000-7f63a90c2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> 7f63a90c2000-7f63a92e2000 r--s 00000000 08:21 9442763                    /ssd/cassandra/data/iss/feedback_history-d32bc7e048c011e49b989bc3e8a5a440/iss-feedback_history-tmplink-ka-328671-Index.db
(deleted)
> {quote}
> {quote}
> $ sudo grep deleted /proc/15724/maps | wc -l
> 640118
> {quote}
> {quote}
> $ sudo grep -v deleted /proc/15724/maps | wc -l
> 303340
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message