cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mck SembWever (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3427) CompressionMetadata is not shared across threads, we create a new one for each read
Date Mon, 14 Nov 2011 20:06:52 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149861#comment-13149861
] 

Mck SembWever commented on CASSANDRA-3427:
------------------------------------------

startup log with debug{noformat}INFO  48:34,688 DatabaseDescriptor: Loading settings from
file:/iad/finn/countstatistics/conf/cassandra-prod.yaml
INFO  48:34,782 DatabaseDescriptor: DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO  48:34,792 DatabaseDescriptor: Global memtable threshold is enabled at 512MB
INFO  48:34,890 AbstractCassandraDaemon: JVM vendor/version: Java HotSpot(TM) 64-Bit Server
VM/1.6.0_24
INFO  48:34,891 AbstractCassandraDaemon: Heap size: 760414208/8506048512
INFO  48:34,891 AbstractCassandraDaemon: Classpath: /iad/finn/countstatistics/jar/countstatistics.jar:/iad/common/apps/cassandra/lib/jamm-0.2.5.jar
INFO  48:37,158 CLibrary: JNA mlockall successful
INFO  48:37,879 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/Versions-h-42
(256 bytes)
INFO  48:37,879 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/Versions-h-41
(256 bytes)
INFO  48:37,879 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/Versions-h-40
(256 bytes)
INFO  48:37,959 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/IndexInfo-h-3
(223 bytes)
INFO  48:38,001 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/Schema-h-15
(34257 bytes)
INFO  48:38,045 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/Migrations-h-15
(78524 bytes)
INFO  48:38,096 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/LocationInfo-h-150
(80 bytes)
INFO  48:38,096 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/LocationInfo-h-149
(628 bytes)
INFO  48:38,096 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/system/LocationInfo-h-151
(163 bytes)
INFO  48:38,192 DatabaseDescriptor: Loading schema version 1940c630-0be4-11e1-0000-d1695892b1ff
INFO  51:35,136 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191473
(38646535 bytes)
INFO  51:35,136 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-190467
(2284524668 bytes)
INFO  51:35,136 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191469
(254927460 bytes)
INFO  51:35,136 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191475
(30477008 bytes)
INFO  51:35,136 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-114136
(156044360682 bytes)
INFO  51:35,137 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191294
(4585008988 bytes)
INFO  51:35,137 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-190415
(15857295280 bytes)
INFO  51:35,137 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-183183
(196289440978 bytes)
INFO  51:35,137 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191472
(1346076 bytes)
INFO  51:35,137 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-190736
(4626053255 bytes)
INFO  51:35,137 SSTableReader: Opening /iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191435
(1188223188 bytes)
INFO  51:35,187 CompressionMetadata: Allocating chunks index for 5745 chunks for uncompressed
size of 1470519 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191472-CompressionInfo.db)
INFO  51:35,421 CompressionMetadata: Allocating chunks index for 129646 chunks for uncompressed
size of 33189311 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191475-CompressionInfo.db)
INFO  51:35,544 CompressionMetadata: Allocating chunks index for 165602 chunks for uncompressed
size of 42393918 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191473-CompressionInfo.db)
INFO  51:37,171 CompressionMetadata: Allocating chunks index for 1091377 chunks for uncompressed
size of 279392485 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191469-CompressionInfo.db)
INFO  51:41,148 CompressionMetadata: Allocating chunks index for 5086138 chunks for uncompressed
size of 1302051278 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191435-CompressionInfo.db)
INFO  51:46,351 CompressionMetadata: Allocating chunks index for 9766541 chunks for uncompressed
size of 2500234376 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-190467-CompressionInfo.db)
INFO  51:56,717 CompressionMetadata: Allocating chunks index for 19828434 chunks for uncompressed
size of 5076078986 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-190736-CompressionInfo.db)
INFO  51:56,897 CompressionMetadata: Allocating chunks index for 19626358 chunks for uncompressed
size of 5024347477 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-191294-CompressionInfo.db)
INFO  52:21,670 CompressionMetadata: Allocating chunks index for 67865822 chunks for uncompressed
size of 17373650297 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-190415-CompressionInfo.db)
INFO  55:55,920 CompressionMetadata: Allocating chunks index for 666981588 chunks for uncompressed
size of 170747286320 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-114136-CompressionInfo.db)
INFO  56:49,620 CompressionMetadata: Allocating chunks index for 840404671 chunks for uncompressed
size of 215143595584 (/iad/finn/countstatistics/cassandra-data/countstatisticsCount/thrift_no_finntech_countstats_count_Count_neg8589045746818385983-h-183183-CompressionInfo.db)
ERROR 57:51,112 AbstractCassandraDaemon: Fatal exception in thread Thread[SSTableBatchOpen:8,5,main]
java.lang.OutOfMemoryError: Java heap space
	at org.apache.cassandra.utils.BigLongArray.<init>(BigLongArray.java:53)
	at org.apache.cassandra.utils.BigLongArray.<init>(BigLongArray.java:39)
	at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:127)
        ...{noformat}
                
> CompressionMetadata is not shared across threads, we create a new one for each read
> -----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-3427
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3427
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Sylvain Lebresne
>            Assignee: Sylvain Lebresne
>              Labels: compression
>             Fix For: 1.0.2
>
>         Attachments: 0001-debugging.patch, 3427.patch, 3427_v2.patch, CASSANDRA-3427.patch,
jmx_jvm_memory-month.png, jmx_jvm_memory-week.png
>
>
> The CompressionMetada holds the compressed block offsets in memory. Without being absolutely
huge, this is still of non-negligible size as soon as you have a bit of data in the DB. Reallocating
this for each read is a very bad idea.
> Note that this only affect range queries, since "normal" queries uses CompressedSegmentedFile
that does reuse a unique CompressionMetadata instance.
> ( Background: http://thread.gmane.org/gmane.comp.db.cassandra.user/21362 )

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message