incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Forkalsrud <eforkals...@cj.com>
Subject Re: FW: Very slow batch insert using version 0.7.2
Date Fri, 11 Mar 2011 02:40:30 GMT

I see the same behavior with smaller batch sizes.  It appears to happen 
when starting Cassandra with the defaults on relatively large systems.  
Attached is a script I created to reproduce the problem. (usage: 
mutate.sh  /path/to/apache-cassandra-0.7.3-bin.tar.gz)   It extracts a 
stock cassandra distribution to a temp dir and starts it up, (single 
node) then creates a keyspace with a column family and does a batch 
mutate to insert 5000 columns.

When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM)  it 
flushes one Memtable with 5000 operations
When I run it on a server  (RHEL5, 64-bit, 16 cores, 96GB RAM) it 
flushes 100 Memtables with anywhere between 1 operation and 359 
operations (35 bytes and 12499 bytes)

I'm guessing I can override the JVM memory parameters to avoid the 
frequent flushing on the server, but I haven't experimented with that 
yet.  The only difference in the effective command line between the 
laptop and server is "-Xms3932M -Xmx3932M -Xmn400M" on the laptop and 
"-Xms48334M -Xmx48334M -Xmn1600M" on the server.



--
Erik Forkalsrud
Commission Junction


On 03/10/2011 09:18 AM, Ryan King wrote:
> Why use such a large batch size?
>
> -ryan
>
> On Thu, Mar 10, 2011 at 6:31 AM, Desimpel, Ignace
> <Ignace.Desimpel@nuance.com>  wrote:
>>
>> Hello,
>>
>> I had a demo application with embedded cassandra version 0.6.x, inserting
>> about 120 K  row mutations in one call.
>>
>> In version 0.6.x that usually took about 5 seconds, and I could repeat this
>> step adding each time the same amount of data.
>>
>> Running on a single CPU computer, single hard disk, XP 32 bit OS, 1G memory
>>
>> I tested this again on CentOS 64 bit OS, 6G memory, different settings of
>> memtable_throughput_in_mb and memtable_operations_in_millions.
>>
>> Also tried version 0.7.3. Also the same behavior.
>>
>>
>>
>> Now with version 0.7.2 the call returns with a timeout exception even using
>> a timeout of 120000 (2 minutes). I see the CPU time going to 100%, a lot of
>> disk writing ( giga bytes), a lot of log messages  about compacting,
>> flushing, commitlog, …
>>
>>
>>
>> Below you can find some information using the nodetool at start of the batch
>> mutation and also after 14 minutes. The MutationStage is clearly showing how
>> slow the system handles the row mutations.
>>
>>
>>
>> Attached : Cassandra.yaml with at end the description of my database
>> structure using yaml
>>
>> Attached : log file with cassandra output.
>>
>>
>>
>> Any idea what I could be doing wrong?
>>
>>
>>
>> Regards,
>>
>>
>>
>> Ignace Desimpel
>>
>>
>>
>> ignace.desimpel@nuance.com
>>
>>
>>
>> At start of the insert (after inserting 124360 row mutations) I get the
>> following info from the nodetool :
>>
>>
>>
>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>>
>> Starting NodeTool
>>
>> 34035877798200531112672274220979640561
>>
>> Gossip active    : true
>>
>> Load             : 5.49 MB
>>
>> Generation No    : 1299502115
>>
>> Uptime (seconds) : 1152
>>
>> Heap Memory (MB) : 179,84 / 1196,81
>>
>>
>>
>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>>
>> Starting NodeTool
>>
>> Pool Name                    Active   Pending      Completed
>>
>> ReadStage                         0         0          40637
>>
>> RequestResponseStage              0         0             30
>>
>> MutationStage                    32    121679          72149
>>
>> GossipStage                       0         0              0
>>
>> AntiEntropyStage                  0         0              0
>>
>> MigrationStage                    0         0              1
>>
>> MemtablePostFlusher               0         0              6
>>
>> StreamStage                       0         0              0
>>
>> FlushWriter                       0         0              5
>>
>> MiscStage                         0         0              0
>>
>> FlushSorter                       0         0              0
>>
>> InternalResponseStage             0         0              0
>>
>> HintedHandoff                     0         0              0
>>
>>
>>
>> After 14 minutes (timeout exception after 2 minutes : see log file) I get :
>>
>>
>>
>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>>
>> Starting NodeTool
>>
>> 34035877798200531112672274220979640561
>>
>> Gossip active    : true
>>
>> Load             : 10.31 MB
>>
>> Generation No    : 1299502115
>>
>> Uptime (seconds) : 2172
>>
>> Heap Memory (MB) : 733,82 / 1196,81
>>
>>
>>
>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>>
>> Starting NodeTool
>>
>> Pool Name                    Active   Pending      Completed
>>
>> ReadStage                         0         0          40646
>>
>> RequestResponseStage              0         0             30
>>
>> MutationStage                    32    103310          90526
>>
>> GossipStage                       0         0              0
>>
>> AntiEntropyStage                  0         0              0
>>
>> MigrationStage                    0         0              1
>>
>> MemtablePostFlusher               0         0             69
>>
>> StreamStage                       0         0              0
>>
>> FlushWriter                       0         0             68
>>
>> FILEUTILS-DELETE-POOL             0         0             42
>>
>> MiscStage                         0         0              0
>>
>> FlushSorter                       0         0              0
>>
>> InternalResponseStage             0         0              0
>>
>> HintedHandoff                     0         0              0
>>
>>
>>
>>


Mime
View raw message