cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Problem with JVM? concurrent mode failure
Date Wed, 28 Apr 2010 00:05:18 GMT
We're working on this in https://issues.apache.org/jira/browse/CASSANDRA-1014

On Tue, Apr 27, 2010 at 12:28 PM, Daniel Gimenez <danielgs@gmail.com> wrote:
> Hi everyone,
> several days ago I was doing some tests in a Cassandra installation and
> everything was right (few inserts, few deletions, few reads).
>
> Yesterday I started with "stress testing", inserting 10M rows of a
> relational database in a KeySpace. I used 200 threads working in parallel in
> a Large instance (Amazon ec2) to do these inserts.
>
> When the java client I have implemented has inserted almost 5M rows,
> Cassandra server dies returning the message "concurrent mode failure". I
> have monitorized the JVM with Jconsole and the problem is that "old
> generation memory" has crossed the memory that I have specified in the
> JVM_OPTS as you can see in the attached image.
>
> That's my storage-conf.xml
>
> <Storage>
>   <ClusterName>tr_cluster</ClusterName>
>   <AutoBootstrap>false</AutoBootstrap>
>   <Keyspaces>
>     <Keyspace Name="Keyspace1">
>       <ColumnFamily Name="Standard1" CompareWith="BytesType"/>
>       <ColumnFamily Name="Standard2"
>                     CompareWith="UTF8Type"
>                     KeysCached="100%"/>
>       <ColumnFamily Name="StandardByUUID1" CompareWith="TimeUUIDType" />
>       <ColumnFamily Name="Super1"
>                     ColumnType="Super"
>                     CompareWith="BytesType"
>                     CompareSubcolumnsWith="BytesType" />
>       <ColumnFamily Name="Super2"
>                     ColumnType="Super"
>                     CompareWith="UTF8Type"
>                     CompareSubcolumnsWith="UTF8Type"
>                     RowsCached="10000"
>                     KeysCached="50%"
>                     Comment="A column family with supercolumns, whose
column
> and subcolumn names are UTF8 strings"/>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>       <ReplicationFactor>1</ReplicationFactor>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>
>     </Keyspace>
>     <Keyspace Name="tr">
>         <ColumnFamily   ColumnType="Super"
>                         CompareWith="UTF8Type"
>                         CompareSubcolumnsWith="UTF8Type"
>                         Name="Ads"
>                         Coment="List of ads in tr"/>
>         <ColumnFamily CompareWith="TimeUUIDType" Name="States"/>
>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>         <ReplicationFactor>1</ReplicationFactor>
>
> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>   </Keyspace>
>   </Keyspaces>
>
> <Authenticator>org.apache.cassandra.auth.AllowAllAuthenticator</Authenticator>
>   <Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>
>   <InitialToken></InitialToken>
>   <CommitLogDirectory>/opt/space/cassandra/commitlog</CommitLogDirectory>
>   <DataFileDirectories>
>       <DataFileDirectory>/opt/space/cassandra/data</DataFileDirectory>
>   </DataFileDirectories>
>   <Seeds>
>       <Seed>domU-12-31-39-0E-2A-11</Seed>
>   </Seeds>
>   <RpcTimeoutInMillis>10000</RpcTimeoutInMillis>
>   <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
>   <ListenAddress></ListenAddress>
>   <StoragePort>7000</StoragePort>
>   <ThriftAddress></ThriftAddress>
>   <ThriftPort>9160</ThriftPort>
>   <ThriftFramedTransport>false</ThriftFramedTransport>
>   <DiskAccessMode>auto</DiskAccessMode>
>   <RowWarningThresholdInMB>512</RowWarningThresholdInMB>
>   <SlicedBufferSizeInKB>128</SlicedBufferSizeInKB>
>   <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
>   <FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
>   <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
>   <MemtableThroughputInMB>16</MemtableThroughputInMB>
>   <BinaryMemtableThroughputInMB>64</BinaryMemtableThroughputInMB>
>   <MemtableOperationsInMillions>0.1</MemtableOperationsInMillions>
>   <MemtableFlushAfterMinutes>10</MemtableFlushAfterMinutes>
>   <ConcurrentReads>8</ConcurrentReads>
>   <ConcurrentWrites>400</ConcurrentWrites>
>   <CommitLogSync>periodic</CommitLogSync>
>   <CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
>   <GCGraceSeconds>864000</GCGraceSeconds>
> </Storage>
>
>
> And those are the parameters I am using for the JVM:
>
> JVM_OPTS=" \
>         -ea \
>         -Xms2G \
>         -Xmx2G \
>         -XX:NewSize=666m -XX:MaxNewSize=666m \
>         -XX:TargetSurvivorRatio=90 \
>         -XX:+AggressiveOpts \
>         -XX:+UseConcMarkSweepGC
>         -XX:+CMSIncrementalMode \
>         -XX:+CMSIncrementalPacing \
>         -XX:+UseParNewGC \
>         -XX:+CMSParallelRemarkEnabled \
>         -XX:+HeapDumpOnOutOfMemoryError \
>         -XX:HeapDumpPath=/opt/space/ \
>         -XX:SurvivorRatio=128 \
>         -XX:MaxTenuringThreshold=0 \
>         -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
> -XX:PrintCMSStatistics=2 -XX:+PrintGCApplicationStoppedTime \
>         -Djava.rmi.server.hostname=xxx.compute-1.amazonaws.com \
>         -Dcom.sun.management.jmxremote.ssl=false \
>         -Dcom.sun.management.jmxremote.port=9876 \
>         -Dcom.sun.management.jmxremote.authenticate=false"
>
>
> Thank's for all!
> Daniel Gimenez.
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message