incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Gimenez <danie...@gmail.com>
Subject Problem with JVM? concurrent mode failure
Date Tue, 27 Apr 2010 17:28:21 GMT
Hi everyone,
several days ago I was doing some tests in a Cassandra installation and
everything was right (few inserts, few deletions, few reads).

Yesterday I started with "stress testing", inserting 10M rows of a
relational database in a KeySpace. I used 200 threads working in parallel in
a Large instance (Amazon ec2) to do these inserts.

When the java client I have implemented has inserted almost 5M rows,
Cassandra server dies returning the message "concurrent mode failure". I
have monitorized the JVM with Jconsole and the problem is that "old
generation memory" has crossed the memory that I have specified in the
JVM_OPTS as you can see in the attached image.

That's my storage-conf.xml

<Storage>
  <ClusterName>tr_cluster</ClusterName>
  <AutoBootstrap>false</AutoBootstrap>
  <Keyspaces>
    <Keyspace Name="Keyspace1">
      <ColumnFamily Name="Standard1" CompareWith="BytesType"/>
      <ColumnFamily Name="Standard2"
                    CompareWith="UTF8Type"
                    KeysCached="100%"/>
      <ColumnFamily Name="StandardByUUID1" CompareWith="TimeUUIDType" />
      <ColumnFamily Name="Super1"
                    ColumnType="Super"
                    CompareWith="BytesType"
                    CompareSubcolumnsWith="BytesType" />
      <ColumnFamily Name="Super2"
                    ColumnType="Super"
                    CompareWith="UTF8Type"
                    CompareSubcolumnsWith="UTF8Type"
                    RowsCached="10000"
                    KeysCached="50%"
                    Comment="A column family with supercolumns, whose column
and subcolumn names are UTF8 strings"/>

<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
      <ReplicationFactor>1</ReplicationFactor>

<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>

    </Keyspace>
    <Keyspace Name="tr">
        <ColumnFamily   ColumnType="Super"
                        CompareWith="UTF8Type"
                        CompareSubcolumnsWith="UTF8Type"
                        Name="Ads"
                        Coment="List of ads in tr"/>
        <ColumnFamily CompareWith="TimeUUIDType" Name="States"/>

<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
        <ReplicationFactor>1</ReplicationFactor>

<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
  </Keyspace>
  </Keyspaces>

<Authenticator>org.apache.cassandra.auth.AllowAllAuthenticator</Authenticator>
  <Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>
  <InitialToken></InitialToken>
  <CommitLogDirectory>/opt/space/cassandra/commitlog</CommitLogDirectory>
  <DataFileDirectories>
      <DataFileDirectory>/opt/space/cassandra/data</DataFileDirectory>
  </DataFileDirectories>
  <Seeds>
      <Seed>domU-12-31-39-0E-2A-11</Seed>
  </Seeds>
  <RpcTimeoutInMillis>10000</RpcTimeoutInMillis>
  <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
  <ListenAddress></ListenAddress>
  <StoragePort>7000</StoragePort>
  <ThriftAddress></ThriftAddress>
  <ThriftPort>9160</ThriftPort>
  <ThriftFramedTransport>false</ThriftFramedTransport>
  <DiskAccessMode>auto</DiskAccessMode>
  <RowWarningThresholdInMB>512</RowWarningThresholdInMB>
  <SlicedBufferSizeInKB>128</SlicedBufferSizeInKB>
  <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
  <FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
  <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
  <MemtableThroughputInMB>16</MemtableThroughputInMB>
  <BinaryMemtableThroughputInMB>64</BinaryMemtableThroughputInMB>
  <MemtableOperationsInMillions>0.1</MemtableOperationsInMillions>
  <MemtableFlushAfterMinutes>10</MemtableFlushAfterMinutes>
  <ConcurrentReads>8</ConcurrentReads>
  <ConcurrentWrites>400</ConcurrentWrites>
  <CommitLogSync>periodic</CommitLogSync>
  <CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
  <GCGraceSeconds>864000</GCGraceSeconds>
</Storage>


And those are the parameters I am using for the JVM:

JVM_OPTS=" \
        -ea \
        -Xms2G \
        -Xmx2G \
        -XX:NewSize=666m -XX:MaxNewSize=666m \
        -XX:TargetSurvivorRatio=90 \
        -XX:+AggressiveOpts \
        -XX:+UseConcMarkSweepGC
        -XX:+CMSIncrementalMode \
        -XX:+CMSIncrementalPacing \
        -XX:+UseParNewGC \
        -XX:+CMSParallelRemarkEnabled \
        -XX:+HeapDumpOnOutOfMemoryError \
        -XX:HeapDumpPath=/opt/space/ \
        -XX:SurvivorRatio=128 \
        -XX:MaxTenuringThreshold=0 \
        -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:PrintCMSStatistics=2 -XX:+PrintGCApplicationStoppedTime \
        -Djava.rmi.server.hostname=xxx.compute-1.amazonaws.com \
        -Dcom.sun.management.jmxremote.ssl=false \
        -Dcom.sun.management.jmxremote.port=9876 \
        -Dcom.sun.management.jmxremote.authenticate=false"


Thank's for all!
Daniel Gimenez.

Mime
View raw message