cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wade Poziombka (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4314) Index CF tombstones can cause OOM
Date Fri, 08 Jun 2012 19:34:23 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291926#comment-13291926
] 

Wade Poziombka commented on CASSANDRA-4314:
-------------------------------------------

How to compact the index CF?  What is its name?  I've tried a variety of things and can't
seem to find the magic expression.

                
> Index CF tombstones can cause OOM
> ---------------------------------
>
>                 Key: CASSANDRA-4314
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.0
>         Environment: AS5 64, 64 GB ram, 12 core, Intel SSD 
>            Reporter: Wade Poziombka
>            Assignee: Jonathan Ellis
>            Priority: Critical
>             Fix For: 1.0.11, 1.1.2
>
>         Attachments: 2012-06-07-compact.zip, 2012-06-07.zip, 4314-1.0.txt, 4314-1.1.txt,
oom.zip, yourkitsnapshot.png
>
>
> My database (now at 1.0.10) is in a state in which it goes out of memory with hardly
any activity at all.  A key slice nothing more.
> The logs attached are this including verbose gc in stdout.  I started up cassandra and
waited a bit to ensure that it was unperturbed.
> Then (about 15:46) I ran this slice (using Pelops), which in this case should return
NO data.  My client times out and the database goes OOM.
>                   ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in my cluster
>                   Selector s = new Selector(this.pool);
>                   List<IndexExpression> indexExpressions = new ArrayList<IndexExpression>();
>                   IndexExpression e = new IndexExpression(
>                               ByteBuffer.wrap("encryptionSettingsID".getBytes(ASCII)),
IndexOperator.EQ,
>                               ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
>                   indexExpressions.add(e);
>                   IndexClause indexClause = new IndexClause(indexExpressions,
>                               ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
>                   SlicePredicate predicate = new SlicePredicate();
>                   predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
>                         { ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
>                   List<KeySlice> slices = s.getKeySlices(CF_TOKEN, indexClause,
predicate, cl);
> Note that “encryptionSettingsID” is an indexed column.  When this is executed there
should be no columns with the supplied value.
> I suppose I may have some kind of blatant error in this query but it is not obvious to
me.  I’m relatively new to cassandra.
> My key space is defined as follows:
> KsDef(name:TB_UNIT, strategy_class:org.apache.cassandra.locator.SimpleStrategy, strategy_options:{replication_factor=3},

> cf_defs:[
> CfDef(keyspace:TB_UNIT, name:token, column_type:Standard, comparator_type:BytesType,
column_metadata:[ColumnDef(name:70 61 6E 45 6E 63, validation_class:BytesType), ColumnDef(name:63
72 65 61 74 65 54 73, validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74
65, validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate), ColumnDef(name:65
6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44, validation_class:UTF8Type, index_type:KEYS,
index_name:EncryptionSettingsID)], caching:keys_only), 
> CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6, column_type:Standard,
comparator_type:BytesType, caching:keys_only), 
> CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard, comparator_type:BytesType,
column_metadata:[ColumnDef(name:75 73 65 43 6F 75 6E 74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
> ])
> tpstats show pending tasks many minutes after time out:
> [root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> ReadStage                         3         3            107         0              
  0
> RequestResponseStage              0         0             56         0              
  0
> MutationStage                     0         0              6         0              
  0
> ReadRepairStage                   0         0              0         0              
  0
> ReplicateOnWriteStage             0         0              0         0              
  0
> GossipStage                       0         0           2231         0              
  0
> AntiEntropyStage                  0         0              0         0              
  0
> MigrationStage                    0         0              0         0              
  0
> MemtablePostFlusher               0         0              3         0              
  0
> StreamStage                       0         0              0         0              
  0
> FlushWriter                       0         0              3         0              
  0
> MiscStage                         0         0              0         0              
  0
> InternalResponseStage             0         0              0         0              
  0
> HintedHandoff                     0         0              9         0              
  0
> Message type           Dropped
> RANGE_SLICE                  0
> READ_REPAIR                  0
> BINARY                       0
> READ                         0
> MUTATION                     0
> REQUEST_RESPONSE             0
> cfstats:
> Keyspace: keyspace
>         Read Count: 118
>         Read Latency: 0.14722033898305084 ms.
>         Write Count: 0
>         Write Latency: NaN ms.
>         Pending Tasks: 0
>                 Column Family: token
>                 SSTable count: 7
>                 Space used (live): 4745885584
>                 Space used (total): 4745885584
>                 Number of Keys (estimate): 18626048
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 118
>                 Read Latency: 0.147 ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Bloom Filter False Postives: 0
>                 Bloom Filter False Ratio: 0.00000
>                 Bloom Filter Space Used: 55058352
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 150
>                 Compacted row maximum size: 258
>                 Compacted row mean size: 201
>                 Column Family: pan_2fef6478b62242dd94aecaa049b9d7bb
>                 SSTable count: 7
>                 Space used (live): 1987147156
>                 Space used (total): 1987147156
>                 Number of Keys (estimate): 14955264
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 0
>                 Read Latency: NaN ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Bloom Filter False Postives: 0
>                 Bloom Filter False Ratio: 0.00000
>                 Bloom Filter Space Used: 28056224
>                 Key cache: disabled
>                 Row cache: disabled
>                 Compacted row minimum size: 104
>                 Compacted row maximum size: 124
>                 Compacted row mean size: 124
>                 Column Family: counters
>                 SSTable count: 11
>                 Space used (live): 3433469364
>                 Space used (total): 3433469364
>                 Number of Keys (estimate): 21475328
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 0
>                 Read Count: 0
>                 Read Latency: NaN ms.
>                 Write Count: 0
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Bloom Filter False Postives: 0
>                 Bloom Filter False Ratio: 0.00000
>                 Bloom Filter Space Used: 40271696
>                 Key cache capacity: 4652
>                 Key cache size: 4652
>                 Key cache hit rate: NaN
>                 Row cache: disabled
>                 Compacted row minimum size: 125
>                 Compacted row maximum size: 179
>                 Compacted row mean size: 150

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message