Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 548C218D5D for ; Fri, 9 Oct 2015 22:34:28 +0000 (UTC) Received: (qmail 85546 invoked by uid 500); 9 Oct 2015 22:34:24 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 85503 invoked by uid 500); 9 Oct 2015 22:34:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 85493 invoked by uid 99); 9 Oct 2015 22:34:24 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2015 22:34:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D886AC0F46 for ; Fri, 9 Oct 2015 22:34:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.901 X-Spam-Level: ** X-Spam-Status: No, score=2.901 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id ZzSE93g8KH6k for ; Fri, 9 Oct 2015 22:34:12 +0000 (UTC) Received: from mail-qg0-f45.google.com (mail-qg0-f45.google.com [209.85.192.45]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 9A1322315E for ; Fri, 9 Oct 2015 22:34:11 +0000 (UTC) Received: by qgez77 with SMTP id z77so81442694qge.1 for ; Fri, 09 Oct 2015 15:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:message-id:mime-version:subject:date:references :to:in-reply-to; bh=JY2lvpKHwr48R634MQ8k5lmeQZ/wWKSiiFW+2fIZ19U=; b=L4CL+nUkblwQ0dC/oxRCL9+zPRY1X3BZCpYKfJiEErsDs6lqHSM+oz5T0qR6mKTT/g h+dZoF+fG/81mXeFPS4J9tFQ0zPzFjk3pHWRR5wlNUIQXuxWTa9VuR+enr0wHlotVikC njo5/89srCmehL/dH/UOZggTOq4AbCGXBm9gmydWmuW8sQa3GL+fQOeSB6NljI14XnSy g0DMC0kz20+fpnlPX2UtSRelYCmFIkl9YPFw2H0NVdNV+RiR5s3zmlFHmtCRu969HuCI r39DY8O/fEDQVVGj76F4Lc3xWixQc+CsB1Ula7Ovx5Dp+/3vVsvNZ2gHKYPBs8Sqmtu1 wHgw== X-Received: by 10.140.131.135 with SMTP id 129mr19646300qhd.31.1444430045192; Fri, 09 Oct 2015 15:34:05 -0700 (PDT) Received: from nazariocalasmbp.home ([96.57.139.82]) by smtp.gmail.com with ESMTPSA id r5sm1595904qha.48.2015.10.09.15.34.04 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 09 Oct 2015 15:34:04 -0700 (PDT) From: Nazario Parsacala Content-Type: multipart/alternative; boundary="Apple-Mail=_6C9337A3-C481-49D6-B837-0471BD67D293" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3094\)) Subject: Re: Cassandra query degradation with high frequency updated tables. Date: Fri, 9 Oct 2015 18:34:03 -0400 References: <608125AB-23DC-4086-A80B-BAFF86D1423F@gmail.com> <0CC5BE31-FC94-49AC-BC08-5CE5517E66B2@gmail.com> <46FC0CE6-1612-481C-AAF8-C9F5E6090BDC@gmail.com> <2AB81F15-B7BB-4FF7-A62F-D47389714BC7@gmail.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3094) --Apple-Mail=_6C9337A3-C481-49D6-B837-0471BD67D293 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I will send the jstat output later. I have created the ticket: https://issues.apache.org/jira/browse/CASSANDRA-10502 > On Oct 9, 2015, at 5:20 PM, Jonathan Haddad wrote: >=20 > I'd be curious to see GC logs. =20 >=20 > jstat -gccause >=20 > On Fri, Oct 9, 2015 at 2:16 PM Tyler Hobbs > wrote: > Hmm, it seems off to me that the merge step is taking 1 to 2 seconds, = especially when there are only ~500 cells from one sstable and the = memtables. Can you open a ticket = (https://issues.apache.org/jira/browse/CASSANDRA = ) with your schema, = details on your data layout, and these traces? >=20 > On Fri, Oct 9, 2015 at 3:47 PM, Nazario Parsacala = > wrote: >=20 >=20 > So the trace is varying a lot. And does not seem to correlate with the = data return from the client ? Maybe datastax java driver related. ..? = (not likely).. Just checkout the results. >=20 >=20 > Below is the one that I took when from the client (java application) = perspective it was returning data in about 1100 ms. >=20 >=20 >=20 > racing session: 566477c0-6ebc-11e5-9493-9131aba66d63 >=20 > activity = = | = timestamp | source | source_elapsed > = --------------------------------------------------------------------------= --------------------------------------------------------------------------= ------------------------------------------------------------------+-------= ---------------------+---------------+---------------- > = = Execute CQL3 query | = 2015-10-09 15:31:28.700000 | 172.31.17.129 | 0 > Parsing select * from processinfometric_profile where = profilecontext=3D'GENERIC' and id=3D=E2=80=981' and month=3D'Oct' and = day=3D'' and hour=3D'' and minute=3D''; [SharedPool-Worker-1] | = 2015-10-09 15:31:28.701000 | 172.31.17.129 | 101 > = = Preparing statement [SharedPool-Worker-1] | = 2015-10-09 15:31:28.701000 | 172.31.17.129 | 334 > = Executing = single-partition query on processinfometric_profile = [SharedPool-Worker-3] | 2015-10-09 15:31:28.701000 | 172.31.17.129 | = 692 > = = Acquiring sstable references [SharedPool-Worker-3] | = 2015-10-09 15:31:28.701000 | 172.31.17.129 | 713 > = = Merging memtable tombstones [SharedPool-Worker-3] | = 2015-10-09 15:31:28.701000 | 172.31.17.129 | 726 > = = Key cache hit for sstable 209 [SharedPool-Worker-3] | = 2015-10-09 15:31:28.704000 | 172.31.17.129 | 3143 > = = Seeking to partition beginning in data file [SharedPool-Worker-3] | = 2015-10-09 15:31:28.704000 | 172.31.17.129 | 3169 > = = Key cache hit for sstable 208 [SharedPool-Worker-3] | = 2015-10-09 15:31:28.704000 | 172.31.17.129 | 3691 > = = Seeking to partition beginning in data file [SharedPool-Worker-3] | = 2015-10-09 15:31:28.704000 | 172.31.17.129 | 3713 > = Skipped 0/2 = non-slice-intersecting sstables, included 0 due to tombstones = [SharedPool-Worker-3] | 2015-10-09 15:31:28.704000 | 172.31.17.129 | = 3807 > = = Merging data from memtables and 2 sstables [SharedPool-Worker-3] | = 2015-10-09 15:31:28.704000 | 172.31.17.129 | 3818 > = = Read 462 live and 0 tombstone cells [SharedPool-Worker-3] | = 2015-10-09 15:31:29.611000 | 172.31.17.129 | 910723 > = = Request complete | = 2015-10-09 15:31:29.649251 | 172.31.17.129 | 949251 >=20 >=20 >=20 >=20 > Below when this is around 1400 ms . But the trace data seems to look = faster ..? >=20 >=20 >=20 > racing session: 7c591550-6ebf-11e5-9493-9131aba66d63 >=20 > activity = = = | timestamp | source | source_elapsed > = --------------------------------------------------------------------------= --------------------------------------------------------------------------= --------------------------------------------------------------------+-----= -----------------------+---------------+---------------- > = = Execute CQL3 query = | 2015-10-09 15:54:00.869000 | 172.31.17.129 | 0 > Parsing select * from processinfometric_profile where = profilecontext=3D'GENERIC' and id=3D=E2=80=981' and month=3D'Oct' and = day=3D'' and hour=3D'' and minute=3D''; [SharedPool-Worker-133] | = 2015-10-09 15:54:00.869000 | 172.31.17.129 | 122 > = = Preparing statement [SharedPool-Worker-133] = | 2015-10-09 15:54:00.869000 | 172.31.17.129 | 265 > = Executing = single-partition query on processinfometric_profile = [SharedPool-Worker-9] | 2015-10-09 15:54:00.870000 | 172.31.17.129 | = 673 > = = Acquiring sstable references [SharedPool-Worker-9] = | 2015-10-09 15:54:00.870000 | 172.31.17.129 | 695 > = = Merging memtable tombstones [SharedPool-Worker-9] = | 2015-10-09 15:54:00.870000 | 172.31.17.129 | 709 > = = Key cache hit for sstable 242 [SharedPool-Worker-9] = | 2015-10-09 15:54:00.872000 | 172.31.17.129 | 3134 > = = Seeking to partition beginning in data file [SharedPool-Worker-9] = | 2015-10-09 15:54:00.872000 | 172.31.17.129 | 3155 > = Skipped 0/1 = non-slice-intersecting sstables, included 0 due to tombstones = [SharedPool-Worker-9] | 2015-10-09 15:54:00.872000 | 172.31.17.129 | = 3259 > = = Merging data from memtables and 1 sstables [SharedPool-Worker-9] = | 2015-10-09 15:54:00.872000 | 172.31.17.129 | 3270 > = = Read 462 live and 0 tombstone cells [SharedPool-Worker-9] = | 2015-10-09 15:54:02.070000 | 172.31.17.129 | 201640 > = = Request complete = | 2015-10-09 15:54:02.111294 | 172.31.17.129 | 242294 >=20 >=20 >=20 >=20 > This is when it was 1600 ms. >=20 > Tracing session: 9c0ea900-6ec4-11e5-9493-9131aba66d63 >=20 > activity = = = | timestamp | source | source_elapsed > = --------------------------------------------------------------------------= --------------------------------------------------------------------------= --------------------------------------------------------------------+-----= -----------------------+---------------+---------------- > = = Execute CQL3 query = | 2015-10-09 16:30:41.552000 | 172.31.17.129 | 0 > Parsing select * from processinfometric_profile where = profilecontext=3D'GENERIC' and id=3D=E2=80=981' and month=3D'Oct' and = day=3D'' and hour=3D'' and minute=3D''; [SharedPool-Worker-149] | = 2015-10-09 16:30:41.552000 | 172.31.17.129 | 99 > = = Preparing statement [SharedPool-Worker-149] = | 2015-10-09 16:30:41.552000 | 172.31.17.129 | 215 > = Executing = single-partition query on processinfometric_profile = [SharedPool-Worker-164] | 2015-10-09 16:30:41.552000 | 172.31.17.129 | = 507 > = = Acquiring sstable references [SharedPool-Worker-164] = | 2015-10-09 16:30:41.552000 | 172.31.17.129 | 533 > = = Merging memtable tombstones [SharedPool-Worker-164] = | 2015-10-09 16:30:41.552000 | 172.31.17.129 | 551 > = = Key cache hit for sstable 302 [SharedPool-Worker-164] = | 2015-10-09 16:30:41.556000 | 172.31.17.129 | 3875 > = = Seeking to partition beginning in data file [SharedPool-Worker-164] = | 2015-10-09 16:30:41.556000 | 172.31.17.129 | 3902 > = Skipped 0/1 = non-slice-intersecting sstables, included 0 due to tombstones = [SharedPool-Worker-164] | 2015-10-09 16:30:41.556000 | 172.31.17.129 | = 4050 > = = Merging data from memtables and 1 sstables [SharedPool-Worker-164] = | 2015-10-09 16:30:41.556000 | 172.31.17.129 | 4068 > = = Read 468 live and 0 tombstone cells [SharedPool-Worker-164] = | 2015-10-09 16:30:43.269000 | 172.31.17.129 | 717277 > = = Request complete = | 2015-10-09 16:30:43.307559 | 172.31.17.129 | 755559 >=20 >=20 >=20 >=20 > The last one is now at 2300 ms. >=20 >=20 > racing session: 7d89a1e0-6ec6-11e5-9493-9131aba66d63 >=20 > activity = = | = timestamp | source | source_elapsed > = --------------------------------------------------------------------------= --------------------------------------------------------------------------= ------------------------------------------------------------------+-------= ---------------------+---------------+---------------- > = = Execute CQL3 query | = 2015-10-09 16:44:09.342000 | 172.31.17.129 | 0 > Parsing select * from processinfometric_profile where = profilecontext=3D'GENERIC' and id=3D=E2=80=981' and month=3D'Oct' and = day=3D'' and hour=3D'' and minute=3D''; [SharedPool-Worker-1] | = 2015-10-09 16:44:09.342000 | 172.31.17.129 | 87 > = = Preparing statement [SharedPool-Worker-1] | = 2015-10-09 16:44:09.342000 | 172.31.17.129 | 277 > = Executing = single-partition query on processinfometric_profile = [SharedPool-Worker-2] | 2015-10-09 16:44:09.343000 | 172.31.17.129 | = 456 > = = Acquiring sstable references [SharedPool-Worker-2] | = 2015-10-09 16:44:09.343000 | 172.31.17.129 | 473 > = = Merging memtable tombstones [SharedPool-Worker-2] | = 2015-10-09 16:44:09.343000 | 172.31.17.129 | 485 > = = Key cache hit for sstable 328 [SharedPool-Worker-2] | = 2015-10-09 16:44:09.345000 | 172.31.17.129 | 2851 > = = Seeking to partition beginning in data file [SharedPool-Worker-2] | = 2015-10-09 16:44:09.345000 | 172.31.17.129 | 2869 > = Skipped 0/1 = non-slice-intersecting sstables, included 0 due to tombstones = [SharedPool-Worker-2] | 2015-10-09 16:44:09.345000 | 172.31.17.129 | = 3005 > = = Merging data from memtables and 1 sstables [SharedPool-Worker-2] | = 2015-10-09 16:44:09.345001 | 172.31.17.129 | 3017 > = = Read 468 live and 0 tombstone cells [SharedPool-Worker-2] | = 2015-10-09 16:44:11.329000 | 172.31.17.129 | 987011 > = = Request complete | = 2015-10-09 16:44:11.388637 | 172.31.17.129 | 46637 >=20 >=20 >=20 >=20 >> On Oct 9, 2015, at 2:48 PM, Nazario Parsacala > wrote: >>=20 >> Compaction did not help too.=20 >>=20 >>=20 >>=20 >>> On Oct 9, 2015, at 1:01 PM, Nazario Parsacala > wrote: >>>=20 >>> So I upgraded to 2.2.2 and change the compaction strategy from = DateTieredCompactionStrategy to LeveledCompactionStrategy. But the = problem still exists. >>> At the start we were getting responses around 80 to a couple of = hundred of ms. But after 1.5 hours of running, it is now hitting 1447 = ms. I think this will degrade some more as time progresses. I will let = this run a couple of hours more and will also try to force compaction. >>>=20 >>> BTW, with 2.2.2 I am getting the following exceptions. Not sure if = there is already a bug report on this. >>>=20 >>> Caused by: java.io.IOException: Seek position 182054 is not within = mmap segment (seg offs: 0, length: 182054) >>> at = org.apache.cassandra.io.util.ByteBufferDataInput.seek(ByteBufferDataInput.= java:47) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.util.AbstractDataInput.skipBytes(AbstractDataInput= .java:33) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:405) = ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.RowIndexEntry$Serializer.skipPromotedIndex(RowInde= xEntry.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.RowIndexEntry$Serializer.skip(RowIndexEntry.java:1= 55) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigT= ableReader.java:244) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> ... 17 common frames omitted >>> WARN [SharedPool-Worker-42] 2015-10-09 12:54:57,221 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-42,5,main]: {} >>> java.lang.RuntimeException: = org.apache.cassandra.io.sstable.CorruptSSTableException: = java.io.IOException: Seek position 182054 is not within mmap segment = (seg offs: 0, length: 182054) >>> at = org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StoragePro= xy.java:2187) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) = ~[na:1.8.0_60] >>> at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Future= Task.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.2.2.jar:2.2.2] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: = java.io.IOException: Seek position 182054 is not within mmap segment = (seg offs: 0, length: 182054) >>> at = org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigT= ableReader.java:250) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableRe= ader.java:1558) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.sstable.format.big.SSTableSliceIterator.(SST= ableSliceIterator.java:42) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTabl= eReader.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(S= liceQueryFilter.java:246) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryF= ilter.java:62) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.CollationController.collectAllData(CollationContro= ller.java:270) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCo= ntroller.java:64) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyS= tore.java:2004) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStor= e.java:1808) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:360) = ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.j= ava:85) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(St= orageProxy.java:1537) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StoragePro= xy.java:2183) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> ... 4 common frames omitted >>> Caused by: java.io.IOException: Seek position 182054 is not within = mmap segment (seg offs: 0, length: 182054) >>> at = org.apache.cassandra.io.util.ByteBufferDataInput.seek(ByteBufferDataInput.= java:47) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.util.AbstractDataInput.skipBytes(AbstractDataInput= .java:33) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:405) = ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.RowIndexEntry$Serializer.skipPromotedIndex(RowInde= xEntry.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.db.RowIndexEntry$Serializer.skip(RowIndexEntry.java:1= 55) ~[apache-cassandra-2.2.2.jar:2.2.2] >>> at = org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigT= ableReader.java:244) ~[apache-cassandra-2.2.2.jar:2.2.2] >>>=20 >>>=20 >>>=20 >>>=20 >>>> On Oct 9, 2015, at 9:26 AM, Carlos Alonso > wrote: >>>>=20 >>>> Yeah, I was about to suggest the compaction strategy too. Leveled = compaction sounds like a better fit when records are being updated >>>>=20 >>>> Carlos Alonso | Software Engineer | @calonso = >>>>=20 >>>> On 8 October 2015 at 22:35, Tyler Hobbs > wrote: >>>> Upgrade to 2.2.2. Your sstables are probably not compacting due to = CASSANDRA-10270 , = which was fixed in 2.2.2. >>>>=20 >>>> Additionally, you may want to look into using leveled compaction = (http://www.datastax.com/dev/blog/when-to-use-leveled-compaction = ). >>>>=20 >>>> On Thu, Oct 8, 2015 at 4:27 PM, Nazario Parsacala = > wrote: >>>>=20 >>>> Hi, >>>>=20 >>>> so we are developing a system that computes profile of things that = it observes. The observation comes in form of events. Each thing that it = observe has an id and each thing has a set of subthings in it which has = measurement of some kind. Roughly there are about 500 subthings within = each thing. We receive events containing measurements of these 500 = subthings every 10 seconds or so. >>>>=20 >>>> So as we receive events, we read the old profile value, calculate = the new profile based on the new value and save it back. We use the = following schema to hold the profile.=20 >>>>=20 >>>> CREATE TABLE myprofile ( >>>> id text, >>>> month text, >>>> day text, >>>> hour text, >>>> subthings text, >>>> lastvalue double, >>>> count int, >>>> stddev double, >>>> PRIMARY KEY ((id, month, day, hour), subthings) >>>> ) WITH CLUSTERING ORDER BY (subthings ASC) ); >>>>=20 >>>>=20 >>>> This profile will then be use for certain analytics that can use in = the context of the =E2=80=98thing=E2=80=99 or in the context of specific = thing and subthing.=20 >>>>=20 >>>> A profile can be defined as monthly, daily, hourly. So in case of = monthly the month will be set to the current month (i.e. =E2=80=98Oct=E2=80= =99) and the day and hour will be set to empty =E2=80=98=E2=80=99 = string. >>>>=20 >>>>=20 >>>> The problem that we have observed is that over time (actually in = just a matter of hours) we will see a huge degradation of query response = for the monthly profile. At the start it will be respinding in 10-100 = ms and after a couple of hours it will go to 2000-3000 ms . If you leave = it for a couple of days you will start experiencing readtimeouts . The = query is basically just : >>>>=20 >>>> select * from myprofile where id=3D=E2=80=981=E2=80=99 and = month=3D=E2=80=98Oct=E2=80=99 and day=3D=E2=80=98=E2=80=99 and hour=3D=E2=80= =98' >>>>=20 >>>> This will have only about 500 rows or so. >>>>=20 >>>>=20 >>>> I believe that this is cause by the fact there are multiple updates = done to this specific partition. So what do we think can be done to = resolve this ?=20 >>>>=20 >>>> BTW, I am using Cassandra 2.2.1 . And since this is a test , this = is just running on a single node. >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> --=20 >>>> Tyler Hobbs >>>> DataStax >>>>=20 >>>=20 >>=20 >=20 >=20 >=20 >=20 > --=20 > Tyler Hobbs > DataStax --Apple-Mail=_6C9337A3-C481-49D6-B837-0471BD67D293 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
I will send the jstat output later.

I have created the = ticket:

https://issues.apache.org/jira/browse/CASSANDRA-10502
=





On Oct 9, 2015, at 5:20 PM, Jonathan Haddad = <jon@jonhaddad.com> wrote:

I'd be curious to see GC logs.  

jstat -gccause = <pid>

On Fri, Oct 9, 2015 at 2:16 PM Tyler Hobbs <tyler@datastax.com> = wrote:
Hmm, it seems off to = me that the merge step is taking 1 to 2 seconds, especially when there = are only ~500 cells from one sstable and the memtables.  Can you = open a ticket (https://issues.apache.org/jira/browse/CASSANDRA) with = your schema, details on your data layout, and these traces?

On Fri, = Oct 9, 2015 at 3:47 PM, Nazario Parsacala <dodongjuan@gmail.com> wrote:


So the trace is varying a lot. And does not seem to correlate = with the data return from the client ? Maybe datastax java  driver = related. ..? (not likely).. Just checkout the results.


Below is the one that I took when from the client (java = application) perspective it was returning data in  about 1100 = ms.



racing = session: 566477c0-6ebc-11e5-9493-9131aba66d63

 activity           =                     =                     =                     =                     =                     =                     =                     =                     =                     =               | timestamp            =       | source      =   | source_elapsed
---------------------------------------------------------------= --------------------------------------------------------------------------= --------------------------------------------------------------------------= ---+----------------------------+---------------+----------------
     =                     =                     =                     =                     =                     =                     =                     =                     =                     =           Execute CQL3 = query | 2015-10-09 = 15:31:28.700000 | 172.31.17.129 |        =       0
 Parsing select * from = processinfometric_profile where profilecontext=3D'GENERIC' and id=3D=E2=80= =981' and month=3D'Oct' and day=3D'' and hour=3D'' and minute=3D''; = [SharedPool-Worker-1] | 2015-10-09 = 15:31:28.701000 | 172.31.17.129 |          =   101
      =                     =                     =                     =                     =                     =                     =                     =                     =       Preparing statement = [SharedPool-Worker-1] | 2015-10-09 = 15:31:28.701000 | 172.31.17.129 |        =     334
      =                     =                     =                     =                     =                     =                     =     Executing single-partition query on processinfometric_profile = [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.701000 | 172.31.17.129 |        =     692
     =                     =                     =                     =                     =                     =                     =                     =                 =   Acquiring sstable references = [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.701000 | 172.31.17.129 |        =     713
      =                     =                     =                     =                     =                     =                     =                     =                 =   Merging memtable tombstones = [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.701000 | 172.31.17.129 |        =     726
      =                     =                     =                     =                     =                     =                     =                     =                 Key cache hit = for sstable 209 [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.704000 | 172.31.17.129 |       =     3143
      =                     =                     =                     =                     =                     =                     =                     =   Seeking to partition beginning in data file = [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.704000 | 172.31.17.129 |       =     3169
      =                     =                     =                     =                     =                     =                     =                     =                 Key cache hit = for sstable 208 [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.704000 | 172.31.17.129 |       =     3691
      =                     =                     =                     =                     =                     =                     =                     =   Seeking to partition beginning in data file = [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.704000 | 172.31.17.129 |       =     3713
      =                     =                     =                     =                     =                     =             Skipped 0/2 = non-slice-intersecting sstables, included 0 due to tombstones = [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.704000 | 172.31.17.129 |       =     3807
     =                     =                     =                     =                     =                     =                     =                     =     Merging data from memtables and 2 sstables = [SharedPool-Worker-3] | 2015-10-09 = 15:31:28.704000 | 172.31.17.129 |       =     3818
      =                     =                     =                     =                     =                     =                     =                     =           Read 462 live = and 0 tombstone cells [SharedPool-Worker-3] | 2015-10-09 = 15:31:29.611000 | 172.31.17.129 |       =   910723
     =                     =                     =                     =                     =                     =                     =                     =                     =                     =             Request = complete | 2015-10-09 = 15:31:29.649251 | 172.31.17.129 |       =   949251




Below when this is around 1400 ms . But the trace data seems = to look faster ..?



racing session: = 7c591550-6ebf-11e5-9493-9131aba66d63

 activity =                     =                     =                     =                     =                     =                     =                     =                     =                     =                     =       | timestamp            =       | source        | source_elapsed
---------------------------------------------------------------= --------------------------------------------------------------------------= --------------------------------------------------------------------------= -----+----------------------------+---------------+----------------
<= div = style=3D"margin:0px;line-height:normal;font-family:Courier;color:rgb(255,2= 40,165);background-color:rgb(19,119,62)" class=3D"">     =                     =                     =                     =                     =                     =                     =                     =                     =                     =             Execute CQL3 query | 2015-10-09 = 15:54:00.869000 | 172.31.17.129 |          =     0
 Parsing select * from = processinfometric_profile where profilecontext=3D'GENERIC' and id=3D=E2=80= =981' and month=3D'Oct' and day=3D'' and hour=3D'' and minute=3D''; = [SharedPool-Worker-133] | = 2015-10-09 15:54:00.869000 | = 172.31.17.129 |  =           122
      =                     =                     =                     =                     =                     =                     =                     =                     =       Preparing statement [SharedPool-Worker-133] | = 2015-10-09 = 15:54:00.869000 | 172.31.17.129 |          =   265
      =                     =                     =                     =                     =                     =                     =       Executing single-partition query on processinfometric_profile = [SharedPool-Worker-9] | 2015-10-09 15:54:00.870000 | 172.31.17.129 = |            673
     =                     =                     =                     =                     =                     =                     =                     =                     = Acquiring sstable = references [SharedPool-Worker-9] | 2015-10-09 = 15:54:00.870000 | 172.31.17.129 |          =   695
      =                     =                     =                     =                     =                     =                     =                     =                     = Merging memtable = tombstones [SharedPool-Worker-9] | 2015-10-09 = 15:54:00.870000 | 172.31.17.129 |          =   709
      =                     =                     =                     =                     =                     =                     =                     =                   Key cache hit for = sstable 242 [SharedPool-Worker-9] | 2015-10-09 = 15:54:00.872000 | 172.31.17.129 |           = 3134
      =                     =                     =                     =                     =                     =                     =                     =     Seeking to partition beginning in data file = [SharedPool-Worker-9] | 2015-10-09 15:54:00.872000 | 172.31.17.129 = |           3155
      =                     =                     =                     =                     =                     =               Skipped 0/1 = non-slice-intersecting sstables, included 0 due to tombstones = [SharedPool-Worker-9] | 2015-10-09 15:54:00.872000 | 172.31.17.129 = |           3259
     =                     =                     =                     =                     =                     =                     =                     =       Merging data from memtables and 1 sstables = [SharedPool-Worker-9] | 2015-10-09 15:54:00.872000 | 172.31.17.129 = |           3270
      =                     =                     =                     =                     =                     =                     =                     =             Read 462 live and 0 tombstone cells = [SharedPool-Worker-9] | 2015-10-09 15:54:02.070000 | 172.31.17.129 = |         201640
     =                     =                     =                     =                     =                     =                     =                     =                     =                     =               Request = complete | 2015-10-09 15:54:02.111294 | 172.31.17.129 = |         242294




This is when it was 1600 ms.

Tracing session: = 9c0ea900-6ec4-11e5-9493-9131aba66d63

 activity =                     =                     =                     =                     =                     =                     =                     =                     =                     =                     =       | timestamp            =       | source        | source_elapsed
---------------------------------------------------------------= --------------------------------------------------------------------------= --------------------------------------------------------------------------= -----+----------------------------+---------------+----------------
<= div = style=3D"margin:0px;line-height:normal;font-family:Courier;color:rgb(255,2= 40,165);background-color:rgb(19,119,62)" class=3D"">     =                     =                     =                     =                     =                     =                     =                     =                     =                     =             Execute CQL3 query | 2015-10-09 = 16:30:41.552000 | 172.31.17.129 |          =     0
 Parsing select * from = processinfometric_profile where profilecontext=3D'GENERIC' and id=3D=E2=80= =981' and month=3D'Oct' and day=3D'' and hour=3D'' and minute=3D''; = [SharedPool-Worker-149] | = 2015-10-09 16:30:41.552000 | = 172.31.17.129 | =             99
      =                     =                     =                     =                     =                     =                     =                     =                     =       Preparing statement [SharedPool-Worker-149] | = 2015-10-09 = 16:30:41.552000 | 172.31.17.129 |          =   215
      =                     =                     =                     =                     =                     =                     =     Executing single-partition query on processinfometric_profile = [SharedPool-Worker-164] | 2015-10-09 16:30:41.552000 | 172.31.17.129 = |            507
     =                     =                     =                     =                     =                     =                     =                     =                   Acquiring sstable = references [SharedPool-Worker-164] | 2015-10-09 = 16:30:41.552000 | 172.31.17.129 |          =   533
      =                     =                     =                     =                     =                     =                     =                     =                   Merging memtable = tombstones [SharedPool-Worker-164] | 2015-10-09 = 16:30:41.552000 | 172.31.17.129 |          =   551
      =                     =                     =                     =                     =                     =                     =                     =                 Key cache hit for = sstable 302 [SharedPool-Worker-164] | 2015-10-09 = 16:30:41.556000 | 172.31.17.129 |           = 3875
      =                     =                     =                     =                     =                     =                     =                     =   Seeking to = partition beginning in data file [SharedPool-Worker-164] | = 2015-10-09 = 16:30:41.556000 | 172.31.17.129 |           = 3902
      =                     =                     =                     =                     =                     =             Skipped 0/1 non-slice-intersecting sstables, = included 0 due to tombstones [SharedPool-Worker-164] | 2015-10-09 = 16:30:41.556000 | 172.31.17.129 |           = 4050
     =                     =                     =                     =                     =                     =                     =                     =     Merging data from memtables and 1 sstables = [SharedPool-Worker-164] | 2015-10-09 16:30:41.556000 | 172.31.17.129 = |           4068
      =                     =                     =                     =                     =                     =                     =                     =           Read 468 live and 0 tombstone cells = [SharedPool-Worker-164] | 2015-10-09 16:30:43.269000 | 172.31.17.129 = |         717277
     =                     =                     =                     =                     =                     =                     =                     =                     =                     =               Request = complete | 2015-10-09 16:30:43.307559 | 172.31.17.129 = |         755559




The last one is now at 2300 ms.


racing session: = 7d89a1e0-6ec6-11e5-9493-9131aba66d63

 activity =                     =                     =                     =                     =                     =                     =                     =                     =                     =                     =     | timestamp            =       | source        | source_elapsed
---------------------------------------------------------------= --------------------------------------------------------------------------= --------------------------------------------------------------------------= ---+----------------------------+---------------+----------------
     =                     =                     =                     =                     =                     =                     =                     =                     =                     =           Execute CQL3 query | 2015-10-09 = 16:44:09.342000 | 172.31.17.129 |          =     0
 Parsing select * from = processinfometric_profile where profilecontext=3D'GENERIC' and id=3D=E2=80= =981' and month=3D'Oct' and day=3D'' and hour=3D'' and minute=3D''; = [SharedPool-Worker-1] | = 2015-10-09 16:44:09.342000 | = 172.31.17.129 | =             87
      =                     =                     =                     =                     =                     =                     =                     =                     =       Preparing statement [SharedPool-Worker-1] | 2015-10-09 = 16:44:09.342000 | 172.31.17.129 |          =   277
      =                     =                     =                     =                     =                     =                     =     Executing single-partition query on processinfometric_profile = [SharedPool-Worker-2] | 2015-10-09 16:44:09.343000 | 172.31.17.129 = |            456
     =                     =                     =                     =                     =                     =                     =                     =                   Acquiring sstable = references [SharedPool-Worker-2] | 2015-10-09 = 16:44:09.343000 | 172.31.17.129 |          =   473
      =                     =                     =                     =                     =                     =                     =                     =                   Merging memtable = tombstones [SharedPool-Worker-2] | 2015-10-09 = 16:44:09.343000 | 172.31.17.129 |          =   485
      =                     =                     =                     =                     =                     =                     =                     =                 Key cache hit for = sstable 328 [SharedPool-Worker-2] | 2015-10-09 = 16:44:09.345000 | 172.31.17.129 |           = 2851
      =                     =                     =                     =                     =                     =                     =                     =   Seeking to = partition beginning in data file [SharedPool-Worker-2] | = 2015-10-09 = 16:44:09.345000 | 172.31.17.129 |           = 2869
      =                     =                     =                     =                     =                     =             Skipped 0/1 non-slice-intersecting sstables, = included 0 due to tombstones [SharedPool-Worker-2] | 2015-10-09 = 16:44:09.345000 | 172.31.17.129 |           = 3005
     =                     =                     =                     =                     =                     =                     =                     =     Merging data from memtables and 1 sstables = [SharedPool-Worker-2] | 2015-10-09 16:44:09.345001 | 172.31.17.129 = |           3017
      =                     =                     =                     =                     =                     =                     =                     =           Read 468 live and 0 tombstone cells = [SharedPool-Worker-2] | 2015-10-09 16:44:11.329000 | 172.31.17.129 = |         987011
     =                     =                     =                     =                     =                     =                     =                     =                     =                     =             Request complete | 2015-10-09 = 16:44:11.388637 | 172.31.17.129 |          = 46637



On Oct = 9, 2015, at 2:48 PM, Nazario Parsacala <dodongjuan@gmail.com> wrote:

Compaction did not help too. 



On Oct = 9, 2015, at 1:01 PM, Nazario Parsacala <dodongjuan@gmail.com> wrote:

So I upgraded to 2.2.2 and change the compaction strategy = from DateTieredCompactionStrategy to LeveledCompactionStrategy. But the problem still = exists.
At the start we were getting responses = around 80 to a couple of hundred of ms. But after 1.5 hours of running, = it is now hitting 1447 ms. I think this will degrade some more as time = progresses. I will let this run a couple of hours more  and will = also try to force compaction.

BTW, with 2.2.2 I am getting the following = exceptions. Not sure if there is already a bug report on = this.

Caused by: = java.io.IOException: Seek position 182054 is not within mmap segment = (seg offs: 0, length: 182054)
at = org.apache.cassandra.io.util.ByteBufferDataInput.seek(ByteBufferDataInput.= java:47) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.util.AbstractDataInput.skipBytes(AbstractDataInput= .java:33) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:405) = ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.RowIndexEntry$Serializer.skipPromotedIndex(RowInde= xEntry.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.RowIndexEntry$Serializer.skip(RowIndexEntry.java:1= 55) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigT= ableReader.java:244) ~[apache-cassandra-2.2.2.jar:2.2.2]
... 17 common = frames omitted
WARN  = [SharedPool-Worker-42] 2015-10-09 12:54:57,221 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-42,5,main]: {}
java.lang.RuntimeException: = org.apache.cassandra.io.sstable.CorruptSSTableException: = java.io.IOException: Seek position 182054 is not within mmap segment = (seg offs: 0, length: 182054)
at = org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StoragePro= xy.java:2187) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) = ~[na:1.8.0_60]
at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Future= Task.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.2.2.jar:2.2.2]
at = java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: = org.apache.cassandra.io.sstable.CorruptSSTableException: = java.io.IOException: Seek position 182054 is not within mmap segment = (seg offs: 0, length: 182054)
at = org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigT= ableReader.java:250) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableRe= ader.java:1558) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.sstable.format.big.SSTableSliceIterator.<init&g= t;(SSTableSliceIterator.java:42) = ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTabl= eReader.java:75) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(S= liceQueryFilter.java:246) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryF= ilter.java:62) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.CollationController.collectAllData(CollationContro= ller.java:270) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationCo= ntroller.java:64) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyS= tore.java:2004) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStor= e.java:1808) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:360) = ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.j= ava:85) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(St= orageProxy.java:1537) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StoragePro= xy.java:2183) ~[apache-cassandra-2.2.2.jar:2.2.2]
... 4 common = frames omitted
Caused by: = java.io.IOException: Seek position 182054 is not within mmap segment = (seg offs: 0, length: 182054)
at = org.apache.cassandra.io.util.ByteBufferDataInput.seek(ByteBufferDataInput.= java:47) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.util.AbstractDataInput.skipBytes(AbstractDataInput= .java:33) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:405) = ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.RowIndexEntry$Serializer.skipPromotedIndex(RowInde= xEntry.java:164) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.db.RowIndexEntry$Serializer.skip(RowIndexEntry.java:1= 55) ~[apache-cassandra-2.2.2.jar:2.2.2]
at = org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigT= ableReader.java:244) ~[apache-cassandra-2.2.2.jar:2.2.2]




On Oct 9, 2015, at 9:26 AM, = Carlos Alonso <info@mrcalonso.com> wrote:

Yeah, I was about to suggest the = compaction strategy too. Leveled compaction sounds like a better fit = when records are being updated

Carlos = Alonso | Software Engineer | @calonso

On 8 October 2015 at 22:35, = Tyler Hobbs <tyler@datastax.com> wrote:
Upgrade to 2.2.2.  Your sstables are = probably not compacting due to CASSANDRA-10270, which was fixed in = 2.2.2.

Additionally, you may want to = look into using leveled compaction (http://www.datastax.com/dev/blog/when-to-use-leveled-compaction= ).

On Thu, Oct 8, = 2015 at 4:27 PM, Nazario Parsacala <dodongjuan@gmail.com> wrote:

Hi,

so we are developing a system that = computes profile of things that it observes. The observation comes in = form of events. Each thing that it observe has an id and each thing has = a set of subthings in it which has measurement of some kind. Roughly = there are about 500 subthings within each thing. We receive events = containing measurements of these 500 subthings every 10 seconds or = so.

So as we = receive events, we  read the old profile value, calculate the new = profile based on the new value and save it back. We use the following = schema to hold the profile. 

CREATE = TABLE myprofile = (
    id text,
    month text,
    day text,
    hour text,
    subthings text,
    lastvalue double,
    count int,
    stddev double,
 PRIMARY = KEY ((id, month, day, = hour), subthings)
) WITH CLUSTERING ORDER BY = (subthings ASC) = );


This profile = will then be use for certain analytics that can use in the context of = the =E2=80=98thing=E2=80=99 or in the context of specific thing and = subthing. 

A profile can be defined as monthly, daily, hourly. So in = case of monthly the month will be set to the current month (i.e. = =E2=80=98Oct=E2=80=99) and the day and hour will be set to empty =E2=80=98= =E2=80=99 string.

The problem = that we have observed is that over time (actually in just a matter of = hours) we will see a huge degradation of query response  for the = monthly profile. At the start it will be respinding in 10-100 ms and = after a couple of hours it will go to 2000-3000 ms . If you leave it for = a couple of days you will start experiencing readtimeouts . The query is = basically just :

select = * from myprofile where id=3D=E2=80=981=E2=80=99 and month=3D=E2=80=98Oct=E2= =80=99 and day=3D=E2=80=98=E2=80=99 and hour=3D=E2=80=98'

This will have only about 500 = rows or so.


I = believe that this is cause by the fact there are multiple updates done = to this specific partition. So what do we think can be done to resolve = this ? 

BTW, I am using Cassandra 2.2.1 . And since this is a test , = this is just running on a single node.







--
Tyler Hobbs
DataStax






--
Tyler Hobbs
DataStax

= --Apple-Mail=_6C9337A3-C481-49D6-B837-0471BD67D293--