incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Svc <jaytechg...@gmail.com>
Subject Re: Insert v/s Update performance
Date Fri, 29 Mar 2013 19:28:36 GMT
Hi Aaron,

Thank you for your input. I have been monitoring my GC activities and
looking at my Heap, it shows pretty linear activities, without any spikes.

When I look at CPU it shows higher utilization while during writes alone. I
also expect hevy read traffic.

When I tried compaction_throughput_* parameter, I obsered that higher
number here in my case gets better CPU utilization and keeps pending
compactions pretty low. How this parameter works? I have 3 nodes and 2 core
each CPU and I have higher writes.

So usually for high *update* and high read situation what parameter we
should consider for tuning?

Thanks,
Jay





On Wed, Mar 27, 2013 at 9:55 PM, aaron morton <aaron@thelastpickle.com>wrote:

> * Check for GC activity in the logs
> * check the volume the commit log is on to see it it's over utilised.
> * check if the dropped messages correlate to compaction, look at the
> compaction_* settings in yaml and consider reducing the throughput.
>
> Like Dean says if you have existing data it will result in more
> compaction. You may be able to get a lot of writes through in a clean new
> cluster, but it also has to work when compaction and repair are running.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 27/03/2013, at 1:43 PM, Jay Svc <jaytechgeek@gmail.com> wrote:
>
> Thanks Dean again!
>
> My use case is high number of reads and writes out of that I am just
> focusing on write now. I thought LCS is a suitable for my situation. I
> tried simillar on STCS and results are same.
>
> I ran nodetool for tpstats and MutationStage pending are very high. At the
> same time the SSTable count and Pending Compaction are high too during my
> updates.
>
> Please find the snapshot of my syslog.
>
> INFO [ScheduledTasks:1] 2013-03-26 15:05:48,560 StatusLogger.java (line
> 116) OpsCenter.rollups86400                    0,0
> INFO [FlushWriter:55] 2013-03-26 15:05:48,608 Memtable.java (line 264)
> Writing Memtable-InventoryPrice@1051586614(11438914/129587272
> serialized/live bytes, 404320 ops)
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,561 MessagingService.java
> (line 658) 2701 MUTATION messages dropped in last 5000ms
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,562 StatusLogger.java (line
> 57) Pool Name                    Active   Pending   Blocked
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,563 StatusLogger.java (line
> 72) ReadStage                         0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,568 StatusLogger.java (line
> 72) RequestResponseStage              0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line
> 72) ReadRepairStage                   0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line
> 72) MutationStage                    32     19967         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line
> 72) ReplicateOnWriteStage             0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line
> 72) GossipStage                       0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line
> 72) AntiEntropyStage                  0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line
> 72) MigrationStage                    0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line
> 72) StreamStage                       0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line
> 72) MemtablePostFlusher               1         1         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line
> 72) FlushWriter                       1         1         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line
> 72) MiscStage                         0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line
> 72) commitlog_archiver                0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line
> 72) InternalResponseStage             0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line
> 72) HintedHandoff                     0         0         0
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line
> 77) CompactionManager                 1        27
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,675 StatusLogger.java (line
> 89) MessagingService                n/a      0,22
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,724 StatusLogger.java (line
> 99) Cache Type                     Size
> Capacity
> KeysToSave                                                         Provider
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line
> 100) KeyCache                     142315
> 2118997
> all
>  INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line
> 106) RowCache                          0
> 0                      all
> org.apache.cassandra.cache.SerializingCacheProvider
> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line
> 113) ColumnFamily                Memtable ops,data
> INFO [ScheduledTasks:1] 2013-03-26 15:0
>
> Thanks,
> Jay
>
>
>
>
> On Tue, Mar 26, 2013 at 7:15 PM, Hiller, Dean <Dean.Hiller@nrel.gov>wrote:
>
>> LCS is generally used for high read vs. write ratio though it sounds like
>> you may be doing a heavy write load instead.  LCS will involve more
>> compactions as you write to the system compared to STCS because LCS is
>> always trying to keep a 1 to 10 ratio between levels.  While LCS will
>> involve more compaction in general(more I/o, more cpu), I am not sure on
>> update vs. insert though From what I understand STCS will happily duplicate
>> rows across SS tables while LCS does not like to do this so as you update
>> you will constantly compact….well, that is my understanding.  Have you
>> tried STCS out at all?  (ps. This is just from what I understand so take
>> with a grain of salt).
>>
>> Also, there are some great tools in the nodetool tool as well so you can
>> get nodetool compactionstats, etc. etc. and see how backlogged you are in
>> pending tasks….how many pending?
>>
>> Later,
>> Dean
>>
>> From: Jay Svc <jaytechgeek@gmail.com<mailto:jaytechgeek@gmail.com>>
>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>> Date: Tuesday, March 26, 2013 6:08 PM
>> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>> Subject: Re: Insert v/s Update performance
>>
>> Thanks Dean,
>>
>> I have used LCS with sstable_size_in_mb of 15. I have also tried bigger
>> sstable_size_in_mb and observed simillar behavior.
>>
>> Does compaction works differently for update v/s Insert? I belive all
>> keys goes to single SST. What other options I should look into?
>>
>> Thanks,
>> Jay
>>
>>
>>
>>
>> On Tue, Mar 26, 2013 at 6:18 PM, Hiller, Dean <Dean.Hiller@nrel.gov
>> <mailto:Dean.Hiller@nrel.gov>> wrote:
>> Most likely compaction kicks in as updates cause duplicated rows in STCS
>> and compaction causes load that may not have been there before(check your
>> logs).  Also, you can increase the number of nodes in your cluster as well
>> to better handle the load.
>>
>> Later,
>> Dean
>>
>> From: Jay Svc <jaytechgeek@gmail.com<mailto:jaytechgeek@gmail.com
>> ><mailto:jaytechgeek@gmail.com<mailto:jaytechgeek@gmail.com>>>
>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org
>> ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
<
>> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
>> Date: Tuesday, March 26, 2013 5:05 PM
>> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>" <
>> user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:
>> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
>> Subject: Insert v/s Update performance
>>
>> Hi Team,
>>
>> I have this 3 node cluster. I am writing data to these node at the rate
>> of 2,000 records/second. What I observed that if I do inserts. (Means
>> records for those keys does not exist, my column family has 0 records to
>> start with) then I have better write performacne, low SSTable count, low
>> pending compaction and write latency is acceptable and CPU utilization on
>> each node between 35% to 85%.
>>
>> When I ran same test but for update this time (means records already
>> exists in Column family with same key), I observed that my SSTable count
>> gone high 3 times. Pending compactions gone high more than 2 times and
>> write latency has gone high too and CPU utilization was almost 92% to 100%.
>>
>> What is a reason of deteriorating Update performance v/s Insert
>> performance. Since this is critical you help is highly appriciated.
>>
>> P.S. I also observed that high number of pending Mutation Stage on my
>> nodetool tpstats.
>>
>> Thanks,
>> Jay
>>
>>
>
>

Mime
View raw message