incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: stress insert benchmark & cassandra hangup/crash
Date Tue, 01 Dec 2009 15:11:06 GMT
If you insert data too quickly you will run out of memory and it will
crash.  0.5 improves this some but it's still possible if you try hard
enough.  So, insert slower until we have backpressure support.

On Tue, Dec 1, 2009 at 8:53 AM, pavel@kreado.com pop <pavel@kreado.com> wrote:
> I did simple insert benchmark in java (see code below). It uses 5
> threads to write records in one CF.
> After some point cassandra hangup, then crashes. I use single node
> cassandra installation with commit log on separate hdd.
>
> Hardware:
> AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
> 4 GB RAM,
> 2xHDD Samsung 1TB 32MB cache 7200RPM
>
> Cassandra config file is standard with my own file paths:
>  <CommitLogDirectory>/media/
> HDD_1000_2/data/cassandra/commitlog</CommitLogDirectory>
>
> <DataFileDirectories>
>
>
>
> <DataFileDirectory>/media/HDD_1000_1/data/cassandra/data</DataFileDirectory>
>
>
>
> </DataFileDirectories>
>
>
>
> <CalloutLocation>/media/HDD_1000_1/data/cassandra/callouts</CalloutLocation>
>
>
>
> <StagingFileDirectory>/media/HDD_1000_1/data/cassandra/staging</StagingFileDirectory>
> and Keyspace 'suggy', CF 'Pages'
>
> At benchmark begin i see lower performance (i think this is because of JIT):
> 10000 time 12.375
> 10000 time 12.577
> 10000 time 12.782
> 10000 time 13.483
> 10000 time 13.475
> 20000 time 4.849
> 20000 time 4.924
> 20000 time 5.437
> 20000 time 4.966
> 20000 time 5.554
> 30000 time 6.22
> 30000 time 5.852
> 30000 time 5.893
> 30000 time 5.965
>
> in some points performance slightly degrades (cassandra is
> writing/sorting/combining memtables):
>
> 110000 time 3.943
> 120000 time 3.3
> 120000 time 3.143
> 120000 time 3.366
> 120000 time 3.437
> 120000 time 7.191
> 130000 time 7.288
> 130000 time 7.456
> 130000 time 6.931
> 130000 time 7.317
> 130000 time 2.985
> 140000 time 1.982
> 140000 time 3.422
> 140000 time 4.511
> 140000 time 4.17
> 140000 time 3.579
> 150000 time 3.807
> 150000 time 3.086
>
> but at some point is performance degrades to unacceptable value:
>
> 350000 time 4.397
> 360000 time 2.944
> 360000 time 5.922
> 360000 time 2.668
> 350000 time 4.251
> 360000 time 4.163
> 370000 time 3.262
> 370000 time 17.045
> 370000 time 17.546
> 360000 time 17.433
> 370000 time 17.259
> 380000 time 17.286
> 380000 time 17.921
> 380000 time 17.557
> 370000 time 32.203
> 380000 time 32.111
> 390000 time 54.099
> 390000 time 98.446
> 390000 time 140.735
> 400000 time 179.168
> 390000 time 201.667
> 380000 time 202.058
> 400000 time 121.411
> 400000 time 79.004
> 390000 time 17.275
> 410000 time 18.713
> 400000 time 18.67
> 410000 time 97.205
> 410000 time 161.88
> 410000 time 167.947
> 420000 time 273.025
> 400000 time 460.794
>
> At this point i don't see any hdd activity, and cassandra thread eat
> 100% of one cpu core.
>
> Last log entries at INFO level:
>
> INFO - Compacting
> [org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-1-Data.db'),org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-2-Data.db'),org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-3-Data.db'),org.apache.cassandra.io.SSTableReader(path='/media/HDD_1000_1/data/cassandra/data/suggy/Pages-4-Data.db')]
> INFO - Pages has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(Pages)@27641662
> INFO - Sorting Memtable(Pages)@27641662
> INFO - Pages has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(Pages)@7518988
> INFO - Sorting Memtable(Pages)@7518988
> INFO - Completed flushing
> /media/HDD_1000_1/data/cassandra/data/suggy/Pages-5-Data.db
> INFO - Writing Memtable(Pages)@17877278
> INFO - Pages has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(Pages)@6467519
> INFO - Sorting Memtable(Pages)@6467519
> INFO - Pages has reached its threshold; switching in a fresh Memtable
> INFO - Enqueuing flush of Memtable(Pages)@25574355
> INFO - Sorting Memtable(Pages)@25574355
> INFO - Completed flushing
> /media/HDD_1000_1/data/cassandra/data/suggy/Pages-7-Data.db
> INFO - Writing Memtable(Pages)@3968125
>
> After long pause cassandra crashes (i use 1GB for java heap):
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid21132.hprof ...
> Heap dump file created [1196864899 bytes in 51.159 secs]
>
> I tried this testcase on three mashines with latest development version
> of cassandra and with 0.4.2. And i got the same problems.
>
> May be there is some problem in my benchmark?
>
> My benchmark code:
>
> import java.util.List;
> import java.io.UnsupportedEncodingException;
>
> import org.apache.thrift.transport.TTransport;
> import org.apache.thrift.transport.TSocket;
> import org.apache.thrift.protocol.TProtocol;
> import org.apache.thrift.protocol.TBinaryProtocol;
> import org.apache.thrift.TException;
> import org.apache.cassandra.service.*;
>
> public class bench implements Runnable
> {
>    public static void main(String[] args)
>    throws InterruptedException, TException, TimedOutException,
> InvalidRequestException, UnavailableException,
> UnsupportedEncodingException, NotFoundException
> {
>
>
>
>
>      bench runnable = new bench();
>      runnable.startseek=10000000;
>      Thread thread = new Thread(runnable);
>      thread.start();
>
>      bench runnable2 = new bench();
>      runnable2.startseek=20000000;
>      Thread thread2 = new Thread(runnable2);
>      thread2.start();
>
>      bench runnable3 = new bench();
>      runnable3.startseek=30000000;
>      Thread thread3 = new Thread(runnable3);
>      thread3.start();
>
>      bench runnable4 = new bench();
>      runnable4.startseek=40000000;
>      Thread thread4 = new Thread(runnable4);
>      thread4.start();
>
>      bench runnable5 = new bench();
>      runnable5.startseek=50000000;
>      Thread thread5 = new Thread(runnable5);
>      thread5.start();
>
>      thread.join();
>      thread2.join();
>      thread3.join();
>      thread4.join();
>      thread5.join();
>    }
>
>    public int startseek;
>
>    public void run() {
>        try {
>          TTransport tr = new TSocket("localhost", 9160);
>          TProtocol proto = new TBinaryProtocol(tr);
>          Cassandra.Client client = new Cassandra.Client(proto);
>          tr.open();
>
>          // insert data
>          long timestamp = System.currentTimeMillis();
>          ColumnPath cp=new ColumnPath("Pages", null,
> "v".getBytes("UTF-8"));
>          int endseek=startseek+1000000;
>          int report_delay=10000;
>          int report_right=0;
>          for(int i=startseek;i<endseek;i++) {
>          if (report_right>=report_delay) {
>            report_right=0;
>            long new_timestamp=System.currentTimeMillis();
>            System.out.println(String.valueOf(i-startseek)+" time
> "+String.valueOf((new_timestamp-timestamp)/1000.0));
>            timestamp=new_timestamp;
>          }
>
>          client.insert("suggy",
>                        String.valueOf(i),
>                        cp,
>
> "================================================================================".getBytes("UTF-8"),
>                        timestamp,
>                        ConsistencyLevel.ZERO);
>          report_right++;
>          }
>
>          tr.close();
>        }
>        catch (TException e) {System.out.println(e);}
>        catch (TimedOutException e) {System.out.println(e);}
>        catch (InvalidRequestException e) {System.out.println(e);}
>        catch (UnavailableException e) {System.out.println(e);}
>        catch (UnsupportedEncodingException e) {System.out.println(e);}
>    }
> }
>
> PS: Sorry for my english
>
>

Mime
View raw message