incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhu Han <schumi....@gmail.com>
Subject Re: FW: Very slow batch insert using version 0.7.2
Date Fri, 11 Mar 2011 12:56:38 GMT
On Fri, Mar 11, 2011 at 10:40 AM, Erik Forkalsrud <eforkalsrud@cj.com>wrote:

>
> I see the same behavior with smaller batch sizes.  It appears to happen
> when starting Cassandra with the defaults on relatively large systems.
>  Attached is a script I created to reproduce the problem. (usage: mutate.sh
>  /path/to/apache-cassandra-0.7.3-bin.tar.gz)   It extracts a stock cassandra
> distribution to a temp dir and starts it up, (single node) then creates a
> keyspace with a column family and does a batch mutate to insert 5000
> columns.
>
> When I run it on my laptop (Fedora 14, 64-bit, 4 cores, 8GB RAM)  it
> flushes one Memtable with 5000 operations
> When I run it on a server  (RHEL5, 64-bit, 16 cores, 96GB RAM) it flushes
> 100 Memtables with anywhere between 1 operation and 359 operations (35 bytes
> and 12499 bytes)
>

What's the settings of commit log flush, periodic or in batch?


>
> I'm guessing I can override the JVM memory parameters to avoid the frequent
> flushing on the server, but I haven't experimented with that yet.  The only
> difference in the effective command line between the laptop and server is
> "-Xms3932M -Xmx3932M -Xmn400M" on the laptop and "-Xms48334M -Xmx48334M
> -Xmn1600M" on the server.
>
>
>
> --
> Erik Forkalsrud
> Commission Junction
>
>
>
> On 03/10/2011 09:18 AM, Ryan King wrote:
>
>> Why use such a large batch size?
>>
>> -ryan
>>
>> On Thu, Mar 10, 2011 at 6:31 AM, Desimpel, Ignace
>> <Ignace.Desimpel@nuance.com>  wrote:
>>
>>>
>>> Hello,
>>>
>>> I had a demo application with embedded cassandra version 0.6.x, inserting
>>> about 120 K  row mutations in one call.
>>>
>>> In version 0.6.x that usually took about 5 seconds, and I could repeat
>>> this
>>> step adding each time the same amount of data.
>>>
>>> Running on a single CPU computer, single hard disk, XP 32 bit OS, 1G
>>> memory
>>>
>>> I tested this again on CentOS 64 bit OS, 6G memory, different settings of
>>> memtable_throughput_in_mb and memtable_operations_in_millions.
>>>
>>> Also tried version 0.7.3. Also the same behavior.
>>>
>>>
>>>
>>> Now with version 0.7.2 the call returns with a timeout exception even
>>> using
>>> a timeout of 120000 (2 minutes). I see the CPU time going to 100%, a lot
>>> of
>>> disk writing ( giga bytes), a lot of log messages  about compacting,
>>> flushing, commitlog, …
>>>
>>>
>>>
>>> Below you can find some information using the nodetool at start of the
>>> batch
>>> mutation and also after 14 minutes. The MutationStage is clearly showing
>>> how
>>> slow the system handles the row mutations.
>>>
>>>
>>>
>>> Attached : Cassandra.yaml with at end the description of my database
>>> structure using yaml
>>>
>>> Attached : log file with cassandra output.
>>>
>>>
>>>
>>> Any idea what I could be doing wrong?
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>> Ignace Desimpel
>>>
>>>
>>>
>>> ignace.desimpel@nuance.com
>>>
>>>
>>>
>>> At start of the insert (after inserting 124360 row mutations) I get the
>>> following info from the nodetool :
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>>>
>>> Starting NodeTool
>>>
>>> 34035877798200531112672274220979640561
>>>
>>> Gossip active    : true
>>>
>>> Load             : 5.49 MB
>>>
>>> Generation No    : 1299502115
>>>
>>> Uptime (seconds) : 1152
>>>
>>> Heap Memory (MB) : 179,84 / 1196,81
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>>>
>>> Starting NodeTool
>>>
>>> Pool Name                    Active   Pending      Completed
>>>
>>> ReadStage                         0         0          40637
>>>
>>> RequestResponseStage              0         0             30
>>>
>>> MutationStage                    32    121679          72149
>>>
>>> GossipStage                       0         0              0
>>>
>>> AntiEntropyStage                  0         0              0
>>>
>>> MigrationStage                    0         0              1
>>>
>>> MemtablePostFlusher               0         0              6
>>>
>>> StreamStage                       0         0              0
>>>
>>> FlushWriter                       0         0              5
>>>
>>> MiscStage                         0         0              0
>>>
>>> FlushSorter                       0         0              0
>>>
>>> InternalResponseStage             0         0              0
>>>
>>> HintedHandoff                     0         0              0
>>>
>>>
>>>
>>> After 14 minutes (timeout exception after 2 minutes : see log file) I get
>>> :
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com info
>>>
>>> Starting NodeTool
>>>
>>> 34035877798200531112672274220979640561
>>>
>>> Gossip active    : true
>>>
>>> Load             : 10.31 MB
>>>
>>> Generation No    : 1299502115
>>>
>>> Uptime (seconds) : 2172
>>>
>>> Heap Memory (MB) : 733,82 / 1196,81
>>>
>>>
>>>
>>> C:\apache-cassandra-07.2\bin>nodetool --host ads.nuance.com tpstats
>>>
>>> Starting NodeTool
>>>
>>> Pool Name                    Active   Pending      Completed
>>>
>>> ReadStage                         0         0          40646
>>>
>>> RequestResponseStage              0         0             30
>>>
>>> MutationStage                    32    103310          90526
>>>
>>> GossipStage                       0         0              0
>>>
>>> AntiEntropyStage                  0         0              0
>>>
>>> MigrationStage                    0         0              1
>>>
>>> MemtablePostFlusher               0         0             69
>>>
>>> StreamStage                       0         0              0
>>>
>>> FlushWriter                       0         0             68
>>>
>>> FILEUTILS-DELETE-POOL             0         0             42
>>>
>>> MiscStage                         0         0              0
>>>
>>> FlushSorter                       0         0              0
>>>
>>> InternalResponseStage             0         0              0
>>>
>>> HintedHandoff                     0         0              0
>>>
>>>
>>>
>>>
>>>
>

Mime
View raw message