Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id EFF0A200C15 for ; Wed, 8 Feb 2017 18:31:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id ED0E1160B5A; Wed, 8 Feb 2017 17:31:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7B2AE160B49 for ; Wed, 8 Feb 2017 18:31:06 +0100 (CET) Received: (qmail 27379 invoked by uid 500); 8 Feb 2017 17:31:04 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 27369 invoked by uid 99); 8 Feb 2017 17:31:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Feb 2017 17:31:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 88DCAC0C2F for ; Wed, 8 Feb 2017 17:31:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id QCXY4-6j3reR for ; Wed, 8 Feb 2017 17:31:02 +0000 (UTC) Received: from mail-ua0-f182.google.com (mail-ua0-f182.google.com [209.85.217.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2BA015F5D3 for ; Wed, 8 Feb 2017 17:31:01 +0000 (UTC) Received: by mail-ua0-f182.google.com with SMTP id y9so115069282uae.2 for ; Wed, 08 Feb 2017 09:31:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=d+PM4bs+mMXGg7D/MkXdlNUs+eQ8wcxs5REXRRt03Cc=; b=rg4HRFXRJ1uvWH9Ytet3eAMH9e1jUEfInOXaELaf2ISvMvG+jhWehcPyMeRWg+4JP2 /Igj7WClclBflOUGqQx397Zz3N6HCtZcRQXFWE3TS6IHHO5hmOZkN99yOrWVOXZzd5CI E7nQKVKBeyS1fHJB4bZm8bJauuYx+Yv5MveclMsNc1N27V77klrUteItwYwdilPLaVrl Ykh2CMxLSQ4ECYeSPDNwv/qx2Bjs5d00hQb/KOVdCjVoIHm2UaTcoorH4XUVTd5zS21z HZgLJm0RaAGpJxJBAAFFZFEBBJpfIWN2zfFePoMBhSE8uT7+Q6tolgU9uh0ZxYBIPTlA Ha1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=d+PM4bs+mMXGg7D/MkXdlNUs+eQ8wcxs5REXRRt03Cc=; b=VP/S7fMWP8249eMjEG+CeHRSf/Lkm8BRM3LOjdTO7AQtI6LJqK+C56inQJHPZSnKI3 duDNJZ47+67Nb7AsQNorKjpbbJygIjwSJHAcfTd8CB6kfiADfwna7DtXVJeb+GO064Nj GNtJQ2n2aD/IfspjKy9PyRY9fo2Z9+WBBIxejEhr+/KXBn3h2IxvinEbco6mfCUFR37x TNHueq6nSPpxgtN+jdoAp9JnD2t2bAQhbw1j1i6fs43RWkzxCEAVOQ61ZKC4oMkEJTmC l9T4Ga1pVkTWko9fTOQ9W0yRn4L4ArmVRPEM0/4SHJ3c4AlOEREFupK2mwFSr6ZC5DPJ MONw== X-Gm-Message-State: AMke39kCDE8/K0s5HRTUc59MgfcllW8Op4ZSgbNJ0wHImwkZgFd603vrwqkqvqxd20KPk6/9flR8470pptP3pg== X-Received: by 10.176.91.155 with SMTP id y27mr11796994uae.151.1486575059881; Wed, 08 Feb 2017 09:30:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.103.48.202 with HTTP; Wed, 8 Feb 2017 09:30:39 -0800 (PST) In-Reply-To: References: From: DuyHai Doan Date: Wed, 8 Feb 2017 18:30:39 +0100 Message-ID: Subject: Re: Time series data model and tombstones To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f403045f8ec86054c60548083af4 archived-at: Wed, 08 Feb 2017 17:31:08 -0000 --f403045f8ec86054c60548083af4 Content-Type: text/plain; charset=UTF-8 Thanks for the update. Good to know that TWCS give you more stability On Wed, Feb 8, 2017 at 6:20 PM, John Sanda wrote: > I wanted to provide a quick update. I was able to patch one of the > environments that is hitting the tombstone problem. It has been running > TWCS for five days now, and things are stable so far. I also had a patch to > the application code to implement date partitioning ready to go, but I > wanted to see how things went with only making the compaction changes. > > On Sun, Jan 29, 2017 at 4:05 PM, DuyHai Doan wrote: > >> In theory, you're right and Cassandra should possibly skip reading cells >> having time < 50. But it's all theory, in practice Cassandra read chunks of >> xxx kilobytes worth of data (don't remember the exact value of xxx, maybe >> 64k or far less) so you may end up reading tombstones. >> >> On Sun, Jan 29, 2017 at 9:24 PM, John Sanda wrote: >> >>> Thanks for the clarification. Let's say I have a partition in an SSTable >>> where the values of time range from 100 to 10 and everything < 50 is >>> expired. If I do a query with time < 100 and time >= 50, are there >>> scenarios in which Cassandra will have to read cells where time < 50? In >>> particular I am wondering if compression might have any affect. >>> >>> On Sun, Jan 29, 2017 at 3:01 PM DuyHai Doan >>> wrote: >>> >>>> "Should the data be sorted by my time column regardless of the >>>> compaction strategy" --> It does >>>> >>>> What I mean is that an old "chunk" of expired data in SSTABLE-12 may be >>>> compacted together with a new chunk of SSTABLE-2 containing fresh data so >>>> in the new resulting SSTable will contain tombstones AND fresh data inside >>>> the same partition, but of course sorted by clustering column "time". >>>> >>>> On Sun, Jan 29, 2017 at 8:55 PM, John Sanda >>>> wrote: >>>> >>>> Since STCS does not sort data based on timestamp, your wide partition >>>> may span over multiple SSTables and inside each SSTable, old data (+ >>>> tombstones) may sit on the same partition as newer data. >>>> >>>> >>>> Should the data be sorted by my time column regardless of the >>>> compaction strategy? I didn't think that the column timestamp came into >>>> play with respect to sorting. I have been able to review some SSTables with >>>> sstablemetadata and I can see that old/expired data is definitely living >>>> with live data. >>>> >>>> >>>> On Sun, Jan 29, 2017 at 2:38 PM, DuyHai Doan >>>> wrote: >>>> >>>> Ok so give it a try with TWCS. Since STCS does not sort data based on >>>> timestamp, your wide partition may span over multiple SSTables and inside >>>> each SSTable, old data (+ tombstones) may sit on the same partition as >>>> newer data. >>>> >>>> When reading by slice, even if you request for fresh data, Cassandra >>>> has to scan over a lot tombstones to fetch the correct range of data thus >>>> your issue >>>> >>>> On Sun, Jan 29, 2017 at 8:19 PM, John Sanda >>>> wrote: >>>> >>>> It was with STCS. It was on a 2.x version before TWCS was available. >>>> >>>> On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan >>>> wrote: >>>> >>>> Did you get this Overwhelming tombstonne behavior with STCS or with >>>> TWCS ? >>>> >>>> If you're using DTCS, beware of its weird behavior and tricky >>>> configuration. >>>> >>>> On Sun, Jan 29, 2017 at 3:52 PM, John Sanda >>>> wrote: >>>> >>>> Your partitioning key is text. If you have multiple entries per id you >>>> are likely hitting older cells that have expired. Descending only affects >>>> how the data is stored on disk, if you have to read the whole partition to >>>> find whichever time you are querying for you could potentially hit >>>> tombstones in other SSTables that contain the same "id". As mentioned >>>> previously, you need to add a time bucket to your partitioning key and >>>> definitely use DTCS/TWCS. >>>> >>>> >>>> As I mentioned previously, the UI only queries recent data, e.g., the >>>> past hour, past two hours, past day, past week. The UI does not query for >>>> anything older than the TTL which is 7 days. My understanding and >>>> expectation was that Cassandra would only scan live cells. The UI is a >>>> separate application that I do not maintain, so I am not 100% certain about >>>> the queries. I have been told that it does not query for anything older >>>> than 7 days. >>>> >>>> On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves >>>> wrote: >>>> >>>> >>>> Your partitioning key is text. If you have multiple entries per id you >>>> are likely hitting older cells that have expired. Descending only affects >>>> how the data is stored on disk, if you have to read the whole partition to >>>> find whichever time you are querying for you could potentially hit >>>> tombstones in other SSTables that contain the same "id". As mentioned >>>> previously, you need to add a time bucket to your partitioning key and >>>> definitely use DTCS/TWCS. >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> - John >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> - John >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >> > > > -- > > - John > --f403045f8ec86054c60548083af4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks for the update. Good to know that TWCS give you mor= e stability

= On Wed, Feb 8, 2017 at 6:20 PM, John Sanda <john.sanda@gmail.com>= ; wrote:
I wanted= to provide a quick update. I was able to patch one of the environments tha= t is hitting the tombstone problem. It has been running TWCS for five days = now, and things are stable so far. I also had a patch to the application co= de to implement date partitioning ready to go, but I wanted to see how thin= gs went with only making the compaction changes.

On Sun, Jan 29,= 2017 at 4:05 PM, DuyHai Doan <doanduyhai@gmail.com> wrot= e:
In theory, you're= right and Cassandra should possibly skip reading cells having time < 50= . But it's all theory, in practice Cassandra read chunks of xxx kilobyt= es worth of data (don't remember the exact value of xxx, maybe 64k or f= ar less) so you may end up reading tombstones.

On Sun, Jan 29, 2017 at 9:24 PM, = John Sanda <john.sanda@gmail.com> wrote:
Thanks for the clarification. Let's say I h= ave a partition in an SSTable where the values of time range from 100 to 10= and everything < 50 is expired. If I do a query with time < 100 and = time >=3D 50, are there scenarios in which Cassandra will have to read c= ells where time < 50? In particular I am wondering if compression might = have any affect.

On Sun, Jan 29, 2017 at 3:01 PM = DuyHai Doan <d= oanduyhai@gmail.com> wrote:
=
"Should the data= be sorted by my time column regardless of the compaction strategy" --= > It does
What I mean is that an old "chunk" of expired data in SSTABLE-= 12 may be compacted together with a new chunk of SSTABLE-2 containing fresh= data so in the new resulting SSTable will contain tombstones AND fresh dat= a inside the same partition, but of course sorted by clustering column &quo= t;time".

On Sun, Jan 29, 2017 at 8:55 PM, John Sanda <jo= hn.sanda@gmail.com> wrote:
Since STCS does not sort dat= a based on timestamp, your wide partition may span over multiple SSTables a= nd inside each SSTable, old data (+ tombstones) may sit on the same partiti= on as newer data.

=
Should the data be sorted by my time column regardless of the = compaction strategy? I didn't think that the column timestamp came into= play with respect to sorting. I have been able to review some SSTables wit= h sstablemetadata and I can see that old/expired data is definitely living = with live data.


On Sun, Jan 29, 2017 at 2:38 PM, DuyHai Doan <doanduyhai@gmail= .com> wrote:
Ok so give it a try with TWCS. Since STCS does not sort data based on = timestamp, your wide partition may span over multiple SSTables and inside e= ach SSTable, old data (+ tombstones) may sit on the same partition as newer= data.

When reading by slice, ev= en if you request for fresh data, Cassandra has to scan over a lot tombston= es to fetch the correct range of data thus your issue

<john.sanda@gmail.com= > wrote:
It was with STCS. It was on a 2.x version before TWCS was a= vailable.=C2=A0

On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan &l= t;doanduy= hai@gmail.com> wrote:
=
Did you get this Overwhelming tombstonne behavior wi= th STCS or with TWCS ?

If you're using DTCS, beware of its weird behavior= and tricky configuration.
On Sun, Jan 29, 2017= at 3:52 PM, John Sanda <john.sanda@gmail.com> wrote:
Your partitioning key is text. If you have multiple entries pe= r id you are likely hitting older cells that have expired. Descending only = affects how the data is stored on disk, if you have to read the whole parti= tion to find whichever time you are querying for you could potentially hit = tombstones in other SSTables that contain the same "id". As menti= oned previously, you need to add a time bucket to your partitioning key and= definitely use DTCS/TWCS.

As I mentioned previous= ly, the UI only queries recent data, e.g., the past hour, past two hours, p= ast day, past week. The UI does not query for anything older than the TTL w= hich is 7 days. My understanding and expectation was that Cassandra would o= nly scan live cells. The UI is a separate application that I do not maintai= n, so I am not 100% certain about the queries. I have been told that it doe= s not query for anything older than 7 days.=C2=A0

On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves <kurt@instaclustr.com> wrote:

Your partitioning key= is text. If you have multiple entries per id you are likely hitting older = cells that have expired. Descending only affects how the data is stored on = disk, if you have to read the whole partition to find whichever time you ar= e querying for you could potentially hit tombstones in other SSTables that = contain the same "id". As mentioned previously, you need to add a= time bucket to your partitioning key and definitely use DTCS/TWCS.





--

- John
<= br class=3D"m_2257466328409035232m_7990262449802582982m_6256947404356912888= gmail_msg">









<= br>



--

- John











<= /div>--

= - John
--f403045f8ec86054c60548083af4--