Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 73EC5F46A for ; Wed, 3 Apr 2013 09:56:03 +0000 (UTC) Received: (qmail 51932 invoked by uid 500); 3 Apr 2013 09:56:00 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 51894 invoked by uid 500); 3 Apr 2013 09:56:00 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 51862 invoked by uid 99); 3 Apr 2013 09:55:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2013 09:55:59 +0000 X-ASF-Spam-Status: No, hits=2.6 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of samuelsson.joel@gmail.com designates 209.85.215.49 as permitted sender) Received: from [209.85.215.49] (HELO mail-la0-f49.google.com) (209.85.215.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Apr 2013 09:55:55 +0000 Received: by mail-la0-f49.google.com with SMTP id fs13so1232232lab.8 for ; Wed, 03 Apr 2013 02:55:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=nQh150fiB5aOVsLepaRI9up90rQQAbyDMqxw6jk+72s=; b=XZ3ZcpE7e2A1R/qn3pIkZr9srbgy0wLCd5kHLsSwGXfFk9rQCrPw6STcGynJMnTvYN qGTjritVKsUmd6CKTx9mBg/0Kqm99eVyxsODPYzPLfRoPeHJouKNy58Uy/4QkTpUc73z 6K6/s9ts4CQpPnKMzjajFnP96oVRjUiEO10J2J35oVRkrJfOKsmkGZomi24spWDzZ+Us 5hzrFeKATXWYMERbZ2XQly9b58LHi+NjBAYFOmMar25WnQnm5PoWU7hMI7LgNKu2TIq4 IGSCStKWy0CbwUZwBr+ERhOngFWlWooF3I+ljWqlH5X76OhddgFOMY6jiaVSEslGR6A+ MPLg== MIME-Version: 1.0 X-Received: by 10.112.146.34 with SMTP id sz2mr756290lbb.4.1364982932740; Wed, 03 Apr 2013 02:55:32 -0700 (PDT) Received: by 10.114.29.70 with HTTP; Wed, 3 Apr 2013 02:55:32 -0700 (PDT) In-Reply-To: References: Date: Wed, 3 Apr 2013 11:55:32 +0200 Message-ID: Subject: Re: Cassandra freezes From: Joel Samuelsson To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b3a85c8d4a01104d971da11 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b3a85c8d4a01104d971da11 Content-Type: text/plain; charset=ISO-8859-1 It seems this problem is back and I am unsure how to solve it. I have a test setup like this: 4 machines run 8 processess each. Each process has 2 threads, 1 for writing 100 000 rows and one for reading another 100 000 rows. Each machine (and process) read and write the exact same rows so it is essentially the same 200 000 rows being read / written. The cassandra cluster is a one node cluster. The first 10-20 runs of the test described above goes smoothly, after that tests take increasingly long time with GC happening almost all the time. Here is my CASS-FREEZE-001 form answers: How big is your JVM heap ? 2GB How many CPUs ? A virtual environment so I can't be perfectly sure but according to their specification, "8 cores". Garbage collection taking long ? ( look for log lines from GCInspector) Yes, these are a few lines seen during 1 test run: INFO [ScheduledTasks:1] 2013-04-03 08:47:40,757 GCInspector.java (line 122) GC for ParNew: 40370 ms for 3 collections, 565045688 used; max is 2038693888 INFO [ScheduledTasks:1] 2013-04-03 08:48:24,720 GCInspector.java (line 122) GC for ParNew: 39840 ms for 2 collections, 614065528 used; max is 2038693888 INFO [ScheduledTasks:1] 2013-04-03 08:49:09,319 GCInspector.java (line 122) GC for ParNew: 37666 ms for 2 collections, 682352952 used; max is 2038693888 INFO [ScheduledTasks:1] 2013-04-03 08:50:02,577 GCInspector.java (line 122) GC for ParNew: 44590 ms for 1 collections, 792861352 used; max is 2038693888 Running out of heap ? ( "heap is .. full" log lines ) Yes. Same run as above: WARN [ScheduledTasks:1] 2013-04-03 08:54:35,108 GCInspector.java (line 139) Heap is 0.8596674853032178 full. You may need to reduce memtable and/or cache sizes. Cassandra is now reducing cache sizes to free up memory. Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2013-04-03 08:54:36,831 GCInspector.java (line 145) Heap is 0.8596674853032178 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically Any tasks backing up / being dropped ? ( nodetool tpstats and ".. dropped in last .. ms" log lines ) Yes. Same run as above: INFO [ScheduledTasks:1] 2013-04-03 08:52:04,943 MessagingService.java (line 673) 31 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2013-04-03 08:52:04,944 MessagingService.java (line 673) 8 READ messages dropped in last 5000ms Are writes really slow? ( nodetool cfhistograms Keyspace ColumnFamily ) Not sure how to interpret the output of nodetool cfhistograms, but here it is (I hope it's fairly readable): Offset SSTables Write Latency Read Latency Row Size Column Count 1 38162520 0 0 0 200000 2 0 22 0 0 0 3 0 1629 0 0 0 4 0 9990 0 0 0 5 0 40169 0 0 0 6 0 161538 0 0 0 7 0 487266 0 0 0 8 0 1096601 0 0 0 10 0 4842978 0 0 0 12 0 7976003 0 0 0 14 0 8673230 0 0 0 17 0 9805730 0 0 0 20 0 5083707 0 0 0 24 0 2541157 0 0 0 29 0 768916 0 0 0 35 0 220440 0 0 0 42 0 112915 0 0 0 50 0 71469 0 0 0 60 0 48909 0 0 0 72 0 50714 0 0 0 86 0 45390 0 0 0 103 0 41975 0 0 0 124 0 40371 0 0 0 149 0 37103 0 0 0 179 0 44631 0 0 0 215 0 43957 0 0 0 258 0 32499 1 0 0 310 0 18446 23056779 0 0 372 0 13113 12580639 0 0 446 0 9862 1017347 0 0 535 0 7480 784506 0 0 642 0 5473 274274 0 0 770 0 4084 56379 0 0 924 0 3046 27979 0 0 1109 0 2205 20206 200000 0 1331 0 1658 16947 0 0 1597 0 1228 16969 0 0 1916 0 896 15848 0 0 2299 0 542 13928 0 0 2759 0 379 11782 0 0 3311 0 326 9761 0 0 3973 0 540 8997 0 0 4768 0 450 7938 0 0 5722 0 270 6552 0 0 6866 0 170 6022 0 0 8239 0 146 6474 0 0 9887 0 166 7969 0 0 11864 0 176 53725 0 0 14237 0 203 10260 0 0 17084 0 255 6827 0 0 20501 0 312 27462 0 0 24601 0 445 11523 0 0 29521 0 736 9904 0 0 35425 0 909 20539 0 0 42510 0 896 14280 0 0 51012 0 904 12443 0 0 61214 0 715 11956 0 0 73457 0 652 10040 0 0 88148 0 474 7992 0 0 105778 0 256 5043 0 0 126934 0 113 2370 0 0 152321 0 75 1189 0 0 182785 0 39 690 0 0 219342 0 44 550 0 0 263210 0 69 551 0 0 315852 0 35 419 0 0 379022 0 35 564 0 0 454826 0 52 504 0 0 545791 0 79 749 0 0 654949 0 61 737 0 0 785939 0 30 399 0 0 943127 0 57 611 0 0 1131752 0 78 488 0 0 1358102 0 23 302 0 0 1629722 0 28 240 0 0 1955666 0 48 294 0 0 2346799 0 28 306 0 0 2816159 0 19 224 0 0 3379391 0 37 212 0 0 4055269 0 24 237 0 0 4866323 0 13 137 0 0 5839588 0 11 99 0 0 7007506 0 4 115 0 0 8409007 0 16 194 0 0 10090808 0 12 156 0 0 12108970 0 12 54 0 0 14530764 0 24 147 0 0 17436917 0 10 114 0 0 20924300 0 3 66 0 0 25109160 0 22 100 0 0+ Some of the write latencies looks really bad, but since they have column count 0 for most, I am not sure what to make of it. How much is lots of data? Lots of data might have been an exaggeration but the test is as described above. Each row read or written is about 1kb in size so each test run generates 4 (machines) * 8 (processes) * 2 (read and write) * 100 000 (rows) * 1kb (row size) = 6250 mb read or written (half of each) Wide or skinny rows? Skinny rows, only a single column is used for each row. Mutations/sec ? The test when run on a freshly rebooted cassandra takes around 390 seconds, and 6400000 rows are read / written during this time period so around 16410 mutations / second (unless I missunderstood the word mutation). Which Compaction Strategy are you using? SizeTieredCompactionStrategy Output of show schema ( cassandra-cli ) for the relevant Keyspace/CF might help as well create column family rawData with column_type = 'Standard' and comparator = 'UTF8Type' and default_validation_class = 'BytesType' and key_validation_class = 'BytesType' and read_repair_chance = 0.1 and dclocal_read_repair_chance = 0.0 and gc_grace = 864000 and min_compaction_threshold = 4 and max_compaction_threshold = 32 and replicate_on_write = true and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' and caching = 'KEYS_ONLY' and column_metadata = [ {column_name : 'created', validation_class : LongType}, {column_name : 'socketHash', validation_class : Int32Type}, {column_name : 'data', validation_class : UTF8Type}, {column_name : 'guid', validation_class : UTF8Type}, {column_name : 'evaluated', validation_class : Int32Type, index_name : 'rawData_evaluated_idx_1', index_type : 0}] and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'}; Only the "data" column is used during the test. What consistency are you doing your writes with ? I am writing with consistency level ONE. What are the values for these settings in cassandra.yaml memtable_total_space_in_mb: No value set in cassandra.yaml, so 1/3 of heap according to documentation (2gb / 3) memtable_flush_writers: No value set in cassandra.yaml, but only one data directory so I assume it is 1. memtable_flush_queue_size: 4 compaction_throughput_mb_per_sec: 16 concurrent_writes: 32 Which version of Cassandra? 1.1.8 Hope this helps you help me :) Best regards, Joel Samuelsson 2013/3/22 Joel Samuelsson > Thanks for the GC suggestion. It seems we didn't have enough CPU power to > handle both the data and GC. Increasing the number of CPU cores made > everything run smoothly at the same load. > > > 2013/3/21 Andras Szerdahelyi > >> Neat! >> Thanks. >> >> From: Sylvain Lebresne >> Reply-To: "user@cassandra.apache.org" >> Date: Thursday 21 March 2013 10:10 >> To: "user@cassandra.apache.org" >> Subject: Re: Cassandra freezes >> >> Prior to 1.2 the index summaries were not saved on disk, and were thus >> computed on startup while the sstable was loaded. In 1.2 they now are saved >> on disk to make startup faster ( >> https://issues.apache.org/jira/browse/CASSANDRA-2392). That being said, >> if the index_interval value used by a summary saved doesn't match the >> current one while the sstable is loaded, the summary is recomputed anyway, >> so restarting a node should always take a new index_interval setting into >> account. >> >> -- >> Sylvain >> >> >> On Thu, Mar 21, 2013 at 9:43 AM, Andras Szerdahelyi < >> andras.szerdahelyi@ignitionone.com> wrote: >> >>> I can not find the reference that notes having to upgradesstables when >>> you >>> change this. I really hope such complex assumptions are not formulating >>> in >>> my head just on their own and there actually exists some kind of reliable >>> reference that clears this up :-) but, >>> >>> # index_interval controls the sampling of entries from the primrary >>> # row index in terms of space versus time. The larger the interval, >>> # the smaller and less effective the sampling will be. In technicial >>> # terms, the interval coresponds to the number of index entries that >>> # are skipped between taking each sample. All the sampled entries >>> # must fit in memory. Generally, a value between 128 and 512 here >>> # coupled with a large key cache size on CFs results in the best trade >>> # offs. This value is not often changed, however if you have many >>> # very small rows (many to an OS page), then increasing this will >>> # often lower memory usage without a impact on performance. >>> >>> it is ( very ) safe to assume the row index is re-built/updated when new >>> sstables are built. >>> Obviously the sample of this index will have to follow this process very >>> closely. >>> >>> It is possible however that the sample itself is not persisted and is >>> built at startup as opposed to *only* when the index changes.( which is >>> what I thought was happening ) >>> It shouldn't be too difficult to verify this, but I'd appreciate if >>> someone who looked at this before could confirm if this is the case. >>> >>> Thanks, >>> Andras >>> >>> On 21/03/13 09:13, "Michal Michalski" wrote: >>> >>> >About index_interval: >>> > >>> >> 1) you have to rebuild stables ( not an issue if you are evaluating, >>> >>doing >>> >> test writes.. Etc, not so much in production ) >>> > >>> >Are you sure of this? As I understand indexes, it's not required because >>> >this parameter defines an interval of in-memory index sample, which is >>> >created during C* startup basing on a primary on-disk index file. The >>> >fact that Heap usage is reduced immediately after C* restart seem to >>> >confirm this, but maybe I miss something? >>> > >>> >M. >>> >>> >> > --047d7b3a85c8d4a01104d971da11 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
It seems this problem is back and I am unsure how to solve= it. I have a test setup like this:
4 machines run 8 processess each. Ea= ch process has 2 threads, 1 for writing 100 000 rows and one for reading an= other 100 000 rows. Each machine (and process) read and write the exact sam= e rows so it is essentially the same 200 000 rows being read / written. The cassandra cluster is a one node cluster.
The first 10-20 runs= of the test described above goes smoothly, after that tests take increasin= gly long time with GC happening almost all the time.

Here is my=A0CASS-FREEZE-001 form answers:

How big is your JVM heap ?=A0
2GB

How many CPUs ?
A virtual environment so I can't be perfectly sure but according t= o their specification, "8 cores".

Garbage collection ta= king long ? ( look for log lines from GCInspector)
Yes, these are a few lines seen during 1 test run:
INFO [Sch= eduledTasks:1] 2013-04-03 08:47:40,757 GCInspector.java (line 122) GC for P= arNew: 40370 ms for 3 collections, 565045688 used; max is 2038693888
INFO [ScheduledTasks:1] 2013-04= -03 08:48:24,720 GCInspector.java (line 122) GC for ParNew: 39840 ms for 2 = collections, 614065528 used; max is 2038693888
INFO [ScheduledTasks:1] 2013-04-03 08:49:0= 9,319 GCInspector.java (line 122) GC for ParNew: 37666 ms for 2 collections= , 682352952 used; max is 2038693888
INFO [ScheduledTasks:1] 2013-04= -03 08:50:02,577 GCInspector.java (line 122) GC for ParNew: 44590 ms for 1 = collections, 792861352 used; max is 2038693888


Running out of heap ? ( "heap is .. full" log lines )
Yes. Same run as above:
WARN [ScheduledTasks:1] 2013-04-03 0= 8:54:35,108 GCInspector.java (line 139) Heap is 0.8596674853032178 full. = =A0You may need to reduce memtable and/or cache sizes. =A0Cassandra is now = reducing cache sizes to free up memory. =A0Adjust reduce_cache_sizes_at thr= eshold in cassandra.yaml if you don't want Cassandra to do this automat= ically
WARN [ScheduledTasks:1] 2013-04-03 08:54:36,831 GCInspector.java= (line 145) Heap is 0.8596674853032178 full. =A0You may need to reduce memt= able and/or cache sizes. =A0Cassandra will now flush up to the two largest = memtables to free up memory. =A0Adjust flush_largest_memtables_at threshold= in cassandra.yaml if you don't want Cassandra to do this automatically=


Any tasks backing up / being dropped ? ( nod= etool tpstats and ".. dropped
in last .. ms" log lines )
Yes. Same run as above:
INFO [ScheduledTasks:1] 2013-04-03 08:52:04,943 MessagingService.j= ava (line 673) 31 MUTATION messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2013-04-03 08:52:04,944 MessagingService= .java (line 673) 8 READ messages dropped in last 5000ms

Are writes really slow? ( nodetool cfhistograms Keyspace ColumnFamily )
Not sure how to interpret the output of nodetool cfhistograms, but here it = is (I hope it's fairly readable):
Offset =A0 =A0 =A0SSTa= bles =A0 =A0 Write Latency =A0 =A0 =A0Read Latency =A0 =A0 =A0 =A0 =A0Row S= ize =A0 =A0 =A0Column Count
1 =A0 =A0 =A0 =A0 =A0 38162520 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0200000
2 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A022 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
3 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A01629 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0
4 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A09990= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
5 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A00 =A0 =A0 =A0 =A0 =A0 =A0 40169 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
6= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0161538 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
7 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0487266 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 0
8 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00= =A0 =A0 =A0 =A0 =A0 1096601 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
10 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 4842978 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 0
12 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 7976003 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0
14 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 8673230 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
17 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 9805730 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 0
20 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 5083707 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0
24 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 = =A0 =A0 =A0 =A0 2541157 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
29 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0768916 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 0
35 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0220440 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 0
42 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A0 =A0 =A0112915 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
50 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 71469 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0
60 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 48909 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 0
72 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 = =A0 =A0 =A0 =A0 =A0 =A0 50714 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
86 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 45390 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0
103 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 41975 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 0
124 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 = =A0 =A0 =A0 =A0 =A0 =A0 40371 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
149 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 37103 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 0
179 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 44631 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 0
215 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 = =A0 =A0 =A0 =A0 =A0 =A0 43957 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
258 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 32499 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 0
310 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 18446 =A0= =A0 =A0 =A0 =A023056779 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0
372 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 13113 =A0 =A0 =A0 =A0 =A012580639 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
446 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A09862 =A0 =A0 =A0 =A0 =A0 101734= 7 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
535 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A07480 = =A0 =A0 =A0 =A0 =A0 =A0784506 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0
642 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 = =A0 =A0 =A0 =A0 =A0 =A05473 =A0 =A0 =A0 =A0 =A0 =A0274274 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
770 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A04084 =A0 =A0 =A0 =A0 = =A0 =A0 56379 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 0
924 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A03046 = =A0 =A0 =A0 =A0 =A0 =A0 27979 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 0
1109 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0= =A0 =A0 =A0 =A0 =A02205 =A0 =A0 =A0 =A0 =A0 =A0 20206 =A0 =A0 =A0 =A0 =A0 = =A0200000 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
1331 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A01658 =A0 =A0 =A0 =A0 =A0 =A0 16947= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
1597 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A01228 =A0= =A0 =A0 =A0 =A0 =A0 16969 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
1916 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 896 =A0 =A0 =A0 =A0 =A0 =A0 15848 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
2299 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 542 =A0 =A0 =A0 =A0 =A0 =A0 1= 3928 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
2759 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 379 =A0= =A0 =A0 =A0 =A0 =A0 11782 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
3311 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 326 =A0 =A0 =A0 =A0 =A0 =A0 =A09761 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
3973 =A0 =A0 =A0= =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 540 =A0 =A0 =A0 =A0 =A0 =A0 = =A08997 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0=
4768 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 450 =A0= =A0 =A0 =A0 =A0 =A0 =A07938 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
5722 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 270 =A0 =A0 =A0 =A0 =A0 =A0 =A06552 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
6866 =A0 =A0 =A0= =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 170 =A0 =A0 =A0 =A0 =A0 =A0 = =A06022 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0=
8239 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 146 =A0= =A0 =A0 =A0 =A0 =A0 =A06474 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
9887 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 166 =A0 =A0 =A0 =A0 =A0 =A0 =A07969 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
11864 =A0 =A0 = =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 176 =A0 =A0 =A0 =A0 =A0 = =A0 53725 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= 0
14237 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 203 =A0= =A0 =A0 =A0 =A0 =A0 10260 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
17084 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 255 =A0 =A0 =A0 =A0 =A0 =A0 =A06827 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
20501 =A0 =A0 = =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 312 =A0 =A0 =A0 =A0 =A0 = =A0 27462 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= 0
24601 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 445 =A0= =A0 =A0 =A0 =A0 =A0 11523 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
29521 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 736 =A0 =A0 =A0 =A0 =A0 =A0 =A09904 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
35425 =A0 =A0 = =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 909 =A0 =A0 =A0 =A0 =A0 = =A0 20539 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= 0
42510 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 896 =A0= =A0 =A0 =A0 =A0 =A0 14280 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
51012 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 904 =A0 =A0 =A0 =A0 =A0 =A0 12443 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
61214 =A0 =A0 =A0= =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 715 =A0 =A0 =A0 =A0 =A0 =A0 1= 1956 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
73457 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 652 =A0= =A0 =A0 =A0 =A0 =A0 10040 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
88148 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 474 =A0 =A0 =A0 =A0 =A0 =A0 =A07992 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
105778 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 256 =A0 =A0 =A0 =A0 =A0 =A0 = =A05043 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0=
126934 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 113 =A0 = =A0 =A0 =A0 =A0 =A0 =A02370 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
152321 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A075 =A0 =A0 =A0 =A0 =A0 =A0 =A01189 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
182785 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A039 =A0 =A0 =A0 =A0 =A0= =A0 =A0 690 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 0
219342 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A044 =A0= =A0 =A0 =A0 =A0 =A0 =A0 550 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
263210 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A069 =A0 =A0 =A0 =A0 =A0 =A0 =A0 551 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
315852 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A035 =A0 =A0 =A0 =A0 =A0= =A0 =A0 419 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 0
379022 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A035 =A0= =A0 =A0 =A0 =A0 =A0 =A0 564 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
454826 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A052 =A0 =A0 =A0 =A0 =A0 =A0 =A0 504 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
545791 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A079 =A0 =A0 =A0 =A0 =A0= =A0 =A0 749 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 0
654949 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A061 =A0= =A0 =A0 =A0 =A0 =A0 =A0 737 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
785939 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A030 =A0 =A0 =A0 =A0 =A0 =A0 =A0 399 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
943127 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A057 =A0 =A0 =A0 =A0 =A0= =A0 =A0 611 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 0
1131752 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A078 =A0= =A0 =A0 =A0 =A0 =A0 =A0 488 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
1358102 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A023 =A0 =A0 =A0 =A0 =A0 =A0 =A0 302 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
1629722 =A0 = =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A028 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 240 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 0
1955666 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A048 =A0= =A0 =A0 =A0 =A0 =A0 =A0 294 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
2346799 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A028 =A0 =A0 =A0 =A0 =A0 =A0 =A0 306 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
2816159 =A0 = =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A019 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 224 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 0
3379391 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A037 =A0= =A0 =A0 =A0 =A0 =A0 =A0 212 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
4055269 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A024 =A0 =A0 =A0 =A0 =A0 =A0 =A0 237 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
4866323 =A0 = =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A013 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 137 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 0
5839588 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A011 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A099 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 0
7007506 =A0 =A0 =A0 =A0 =A0 =A00 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 4 =A0 =A0 =A0 =A0 =A0 =A0 =A0 115 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
8409007 = =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A016 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 194 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 0
10090808 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A012 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 156 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
12108970 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A012 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A054 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
14530764 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A024 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 147 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0=
17436917 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A010 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 114 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 0
20924300 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 3 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A066 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0
25109160 =A0 =A0= =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A022 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 100 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0= +

Some of the write latencies looks really bad, but since= they have column count 0 for most, I am not sure what to make of it.
=



How much is lots of data?=A0<= /span>
Lots of data might have been an=A0exaggeration but the test is as described above. Each row read or writ= ten is about 1kb in size so each test run generates 4 (machines) * 8 (proce= sses) * 2 (read and write) * 100 000 (rows) * 1kb (row size) =3D 6250 mb re= ad or written (half of each)

Wide or skinny rows?=A0
Skinny rows, only a single column is used for each row.
<= div>
Mutations/sec ?
The test when run on a = freshly rebooted cassandra takes around=A0390 seconds, and=A06400000 rows a= re read / written during this time period so around=A016410 mutations / sec= ond (unless I missunderstood the word mutation).

Which Compaction Strategy are you using?=A0
SizeTieredCompactionStrategy

Output of show schema (
cassandra-cli ) for the relevant Keyspace/CF might help as well
create column family rawData
=A0 with column_type =3D= 'Standard'
=A0 and comparator =3D 'UTF8Type'
=A0 and default_va= lidation_class =3D 'BytesType'
=A0 and key_validation_cla= ss =3D 'BytesType'
=A0 and read_repair_chance =3D 0.1
=A0 and dclocal_read_repair_chance =3D 0.0
=A0 and gc_grace =3D 864000
=A0 and min_compaction_threshold= =3D 4
=A0 and max_compaction_threshold =3D 32
=A0 and = replicate_on_write =3D true
=A0 and compaction_strategy =3D '= org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
=A0 and caching =3D 'KEYS_ONLY'
=A0 and column_metad= ata =3D [
=A0 =A0 {column_name : 'created',
=A0= =A0 validation_class : LongType},
=A0 =A0 {column_name : 'so= cketHash',
=A0 =A0 validation_class : Int32Type},
=A0 =A0 {column_name = : 'data',
=A0 =A0 validation_class : UTF8Type},
=A0 =A0 {column_name : 'guid',
=A0 =A0 validation_class = : UTF8Type},
=A0 =A0 {column_name : 'evaluated',
=A0 =A0 validati= on_class : Int32Type,
=A0 =A0 index_name : 'rawData_evaluated= _idx_1',
=A0 =A0 index_type : 0}]
=A0 and compressi= on_options =3D {'sstable_compression' : 'org.apache.cassandra.i= o.compress.SnappyCompressor'};

Only the "data" column is used during t= he test.



What consistency are you doing your writes with ?
I am wr= iting with consistency level ONE.


What are the values for these settings in cassandra.yaml

memtable_total_space_in_mb: No value set in cassandra.yaml, so 1/3 of heap= according to documentation (2gb / 3)
memtable_flush_writers: No value set in cassandra.yaml, but only one data = directory so I assume it is 1.
memtable_flush_queue_size: 4
compaction_throughput_mb_per_se= c: 16
concurrent_writes: 32

<= br style=3D"font-family:arial,sans-serif;font-size:12.727272033691406px">W= hich version of Cassandra?
1.1.8

=

Hope this helps you help me :)
Best regards,
Joel Samuelsson


2013/3/22 Joel Samuelsson <samu= elsson.joel@gmail.com>
Thanks for the GC suggestion. It seems we didn't have = enough CPU power to handle both the data and GC. Increasing the number of C= PU cores made everything run smoothly at the same load.


2013/3/21 Andras Szerdahelyi <andras.szerdahelyi@ignitionone.com>
Neat!
Thanks.

From: Sylvain Lebresne <sylvain@datastax.com= >
Reply-To: "user@cassandra.apache.org&q= uot; <use= r@cassandra.apache.org>
Date: Thursday 21 March 2013 10:10<= br> To: "user@cassandra.apache.org" &= lt;user@cass= andra.apache.org>
Subject: Re: Cassandra freezes

Prior to 1.2 the index summaries were not saved on disk, a= nd were thus computed on startup while the sstable was loaded. In 1.2 they = now are saved on disk to make startup faster (https://issues.apache= .org/jira/browse/CASSANDRA-2392). That being said, if the index_interval value used by a summary saved doesn= 't match the current one while the sstable is loaded, the summary is re= computed anyway, so restarting a node should always take a new index_interv= al setting into account.

--
Sylvain


On Thu, Mar 21, 2013 at 9:43 AM, Andras Szerdahe= lyi <andras.szerdahelyi@ignitionone.com> wrote:
I can not find the reference that notes having to upgradesstables when you<= br> change this. I really hope such complex assumptions are not formulating in<= br> my head just on their own and there actually exists some kind of reliable reference that clears this up :-) but,

# index_interval controls the sampling of entries from the primrary
# row index in terms of space versus time. The larger the interval,
# the smaller and less effective the sampling will be. In technicial
# terms, the interval coresponds to the number of index entries that
# are skipped between taking each sample. All the sampled entries
# must fit in memory. Generally, a value between 128 and 512 here
# coupled with a large key cache size on CFs results in the best trade
# offs. This value is not often changed, however if you have many
# very small rows (many to an OS page), then increasing this will
# often lower memory usage without a impact on performance.

it is ( very ) safe to assume the row index is re-built/updated when new sstables are built.
Obviously the sample of this index will have to follow this process very closely.

It is possible however that the sample itself is not persisted and is
built at startup as opposed to *only* when the index changes.( which is
what I thought was happening )
It shouldn't be too difficult to verify this, but I'd appreciate if=
someone who looked at this before could confirm if this is the case.

Thanks,
Andras

On 21/03/13 09:13, "Michal Michalski" <michalm@opera.com> wrote:

>About index_interval:
>
>> 1) you have to rebuild stables ( not an issue if you are evaluatin= g,
>>doing
>> test writes.. Etc, not so much in production )
>
>Are you sure of this? As I understand indexes, it's not required be= cause
>this parameter defines an interval of in-memory index sample, which is<= br> >created during C* startup basing on a primary on-disk index file. The >fact that Heap usage is reduced immediately after C* restart seem to >confirm this, but maybe I miss something?
>
>M.


--047d7b3a85c8d4a01104d971da11--