Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B75D9DF42 for ; Fri, 14 Sep 2012 08:46:44 +0000 (UTC) Received: (qmail 86340 invoked by uid 500); 14 Sep 2012 08:46:42 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 86251 invoked by uid 500); 14 Sep 2012 08:46:41 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 86240 invoked by uid 99); 14 Sep 2012 08:46:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2012 08:46:41 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a47.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2012 08:46:33 +0000 Received: from homiemail-a47.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTP id ADD5B284057 for ; Fri, 14 Sep 2012 01:46:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=8M9rYyv3Ai5EE+5gYEPGeHbRg2 E=; b=MkoCM0vWamMN88UjvOZpKBSW1wexmmF04OjqGFMmZO4eJ74BdYKQZa7hNn Cs/3ysx2FXLBuna/KDHyQ0MNfwE7ne+QzlQa8oLpfbrzOKMnu8H/gdBybj7VJOgt 2DnSSxvswVbwYrMhzqkH+yhCdPdnYCMvj9T3XLEDy0Q8UOA3Q= Received: from [172.16.1.10] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a47.g.dreamhost.com (Postfix) with ESMTPSA id 0AC2D284055 for ; Fri, 14 Sep 2012 01:46:10 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_AC82158C-00F5-4B10-AF97-81C82E29CA7F" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1486\)) Subject: Re: secondery indexes TTL - strange issues Date: Fri, 14 Sep 2012 20:46:08 +1200 References: <69989DC961D0DB4D805CA94CF460737105FBD4C3@AMSPRD0710MB365.eurprd07.prod.outlook.com> To: user@cassandra.apache.org In-Reply-To: <69989DC961D0DB4D805CA94CF460737105FBD4C3@AMSPRD0710MB365.eurprd07.prod.outlook.com> X-Mailer: Apple Mail (2.1486) --Apple-Mail=_AC82158C-00F5-4B10-AF97-81C82E29CA7F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > INFO [CompactionExecutor:181] 2012-09-13 12:58:37,443 = CompactionTask.java (line > 221) Compacted to = [/var/lib/cassandra/data/Eventstore/EventsByItem/Eventstore-E > ventsByItem.ebi_eventtypeIndex-he-10-Data.db,]. 78,623,000 to 373,348 = (~0% of o > riginal) bytes for 83 keys at 0.000280MB/s. Time: 1,272,883ms. There is a lot of weird things here.=20 It could be levelled compaction compacting an older file for the first = time. But that would be a guess.=20 > Rebuilding the index gives us back the data for a couple of minutes - = then it vanishes again. Are you able to do a test with SiezedTieredCompaction ?=20 Are you able to replicate the problem with a fresh testing CF and some = test Data? If it's only a problem with imported data can you provide a sample of = the failing query ? Any maybe the CF definition ?=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 14/09/2012, at 2:46 AM, Roland Gude wrote: > Hi, > =20 > we have been running a system on Cassandra 0.7 heavily relying on = secondary indexes for columns with TTL. > This has been working like a charm, but we are trying hard to move = forward with Cassandra and are struggling at that point: > =20 > When we put our data into a new cluster (any 1.1.x version =96 = currently 1.1.5) , rebuild indexes and run our system, everything seems = to work good =96 until in some point of time index queries do not return = any data at all anymore (note that the TTL has not yet expired for = several months). > Rebuilding the index gives us back the data for a couple of minutes - = then it vanishes again. > =20 > What seems strange is that compaction apparently is very aggressive: > =20 > INFO [CompactionExecutor:181] 2012-09-13 12:58:37,443 = CompactionTask.java (line > 221) Compacted to = [/var/lib/cassandra/data/Eventstore/EventsByItem/Eventstore-E > ventsByItem.ebi_eventtypeIndex-he-10-Data.db,]. 78,623,000 to 373,348 = (~0% of o > riginal) bytes for 83 keys at 0.000280MB/s. Time: 1,272,883ms. > =20 > =20 > Actually we have switched to LeveledCompaction. Could it be that = leveled compaction does not play nice with indexes? > =20 > =20 --Apple-Mail=_AC82158C-00F5-4B10-AF97-81C82E29CA7F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
INFO [CompactionExecutor:181] 2012-09-13 = 12:58:37,443 CompactionTask.java (line
221) Compacted to = [/var/lib/cassandra/data/Eventstore/EventsByItem/Eventstore-E
ventsByItem.ebi_eventtypeIndex-he-10-Data.db,].  = 78,623,000 to 373,348 (~0% of o
riginal) bytes for 83 keys = at 0.000280MB/s.  Time: = 1,272,883ms.
There is a lot of = weird things here. 
It could be levelled compaction compacting = an older file for the first time. But that would be a = guess. 

Rebuilding the index gives us back the data for a couple = of minutes - then it vanishes = again.
Are you able to do a test with SiezedTieredCompaction = ? 

Are you able to replicate the problem with a fresh = testing CF and some test Data?

If it's only a problem with imported data can you provide = a sample of the failing query ? Any maybe the CF definition = ? 

Cheers


http://www.thelastpickle.com

On 14/09/2012, at 2:46 AM, Roland Gude <roland.gude@ez.no> = wrote:

Hi,
 
we have been running a system on Cassandra 0.7 heavily = relying on secondary indexes for columns with = TTL.
This has been working like a charm, but we are trying = hard to move forward with Cassandra and are struggling at that = point:
 
When we put our data into a new cluster (any 1.1.x = version =96 currently 1.1.5) , rebuild indexes and run our system, = everything seems to work good =96 until in some point of time index = queries do not return any data at all anymore (note that the TTL has not = yet expired for several months).
Rebuilding the index gives = us back the data for a couple of minutes - then it vanishes = again.
 
What seems strange is that compaction apparently is very = aggressive:
 
INFO = [CompactionExecutor:181] 2012-09-13 12:58:37,443 CompactionTask.java = (line
221) Compacted to = [/var/lib/cassandra/data/Eventstore/EventsByItem/Eventstore-E
ventsByItem.ebi_eventtypeIndex-he-10-Data.db,].  = 78,623,000 to 373,348 (~0% of o
riginal) bytes for 83 keys = at 0.000280MB/s.  Time: 1,272,883ms.
 
 
Actually we have switched to = LeveledCompaction. Could it be that leveled compaction does not play = nice with indexes?
 
 

= --Apple-Mail=_AC82158C-00F5-4B10-AF97-81C82E29CA7F--