cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milan Majercik (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-13011) heap exhaustion when cleaning table with wide partitions and a secondary index attached to it
Date Wed, 07 Dec 2016 07:43:59 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-13011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Milan Majercik updated CASSANDRA-13011:
---------------------------------------
    Description: 
We have a table with rather wide partitions and a secondary index attached to it. When tried
to clean unused data on a node after expansion of our cluster via issuing {{nodetool cleanup}}
command we observed a heap exhaustion issue. The culprit appears to be in method {{org.apache.cassandra.db.compaction.CompactionManager.CleanupStrategy.Full.cleanup}}
as it tries to remove related secondary index entries. The method first populates a list will
all cells belonging to the given partition...
{code:java}
                while (row.hasNext())
                {
                    OnDiskAtom column = row.next();

                    if (column instanceof Cell && cfs.indexManager.indexes((Cell)
column))
                    {
                        if (indexedColumnsInRow == null)
                            indexedColumnsInRow = new ArrayList<>();

                        indexedColumnsInRow.add((Cell) column);
                    }
                }
{code}

... and then submits it to the index manager for removal.
{code:java}
                    // acquire memtable lock here because secondary index deletion may cause
a race. See CASSANDRA-3712
                    try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start())
                    {
                        cfs.indexManager.deleteFromIndexes(row.getKey(), indexedColumnsInRow,
opGroup);
                    }
{code}

After imposing a limit on array size and implementing some sort of pagination the cleanup
worked fine.

> heap exhaustion when cleaning table with wide partitions and a secondary index attached
to it
> ---------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-13011
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13011
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Compaction
>            Reporter: Milan Majercik
>
> We have a table with rather wide partitions and a secondary index attached to it. When
tried to clean unused data on a node after expansion of our cluster via issuing {{nodetool
cleanup}} command we observed a heap exhaustion issue. The culprit appears to be in method
{{org.apache.cassandra.db.compaction.CompactionManager.CleanupStrategy.Full.cleanup}} as it
tries to remove related secondary index entries. The method first populates a list will all
cells belonging to the given partition...
> {code:java}
>                 while (row.hasNext())
>                 {
>                     OnDiskAtom column = row.next();
>                     if (column instanceof Cell && cfs.indexManager.indexes((Cell)
column))
>                     {
>                         if (indexedColumnsInRow == null)
>                             indexedColumnsInRow = new ArrayList<>();
>                         indexedColumnsInRow.add((Cell) column);
>                     }
>                 }
> {code}
> ... and then submits it to the index manager for removal.
> {code:java}
>                     // acquire memtable lock here because secondary index deletion may
cause a race. See CASSANDRA-3712
>                     try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start())
>                     {
>                         cfs.indexManager.deleteFromIndexes(row.getKey(), indexedColumnsInRow,
opGroup);
>                     }
> {code}
> After imposing a limit on array size and implementing some sort of pagination the cleanup
worked fine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message