Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <CAG7ZRth4yakmFw8qz=ad8LLsvWpbETCteh7Usa7BqJgq79JR=g@mail.gmail.com>
References: <CAG7ZRtjs-xnWmbCzzWa2dXPcxekmkS-ogUfbFYK872tbeHiMRg@mail.gmail.com>
 <CAG7ZRth4yakmFw8qz=ad8LLsvWpbETCteh7Usa7BqJgq79JR=g@mail.gmail.com>
From: Alain RODRIGUEZ <arodrime@gmail.com>
Date: Sun, 18 Dec 2016 17:31:01 +0100
Message-ID: <CA+VSrLpL9Cpm84z=zHH=jonMTtW6_NtbbtfKDMcykqXzM4Hcxw@mail.gmail.com>
Subject: Re: Schema help required
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0b321a59eb580543f155bc
archived-at: Sun, 18 Dec 2016 16:31:36 -0000

--94eb2c0b321a59eb580543f155bc
Content-Type: text/plain; charset=UTF-8

Hi Sagar,


> But this is a known anti pattern to not use Cassandra as a queue causing
> tombstones etc.
> But I could not think of any other way. Does anyone have any other
> suggestion so as to not delete after a pair is created


I believe you could try using a fixed TTL (defined at the table level for
example), then use a TWCS compaction strategy and compactions options that
would efficiently manage tombstones. A colleague at The Last Pickle just
wrote an article about TWCS:
http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html, and there is a
lot more information around, including a talk this year at the summit from
Jeff who contributed with TWCS to Apache Cassandra:
https://www.youtube.com/watch?v=PWtekUWCIaw.

Also using a time buckets in the partition key could help making sure
tombstones will be correctly removed and are not being scanned when
requesting new data. Yet do not use *only* a time bucket in the partition
key as it would lead to hotspots. For a given date, only one node (+
replicas) would handle the write / read load.

So using "day + something else" as a partition key and "TWCS + Fixed TTLs"
*could* be a good way to move forward.

I would give it a try with the cassandra-stress tool that is shipped
alongside Apache Cassandra and allows the use of a user defined schema.

C*heers,

Alain

2016-12-17 21:02 GMT+01:00 Sagar Jambhulkar <sagar.jambhulkar@gmail.com>:

> Hi,
> Needed a suggestion for a schema query. I want to build a reconciliation
> using Cassandra. Basically two or more systems send message to a
> reconciliation process. The reconciliation process first does a level one
> match of id's and than does complete comparison of messages.
>
> The best I could think of is a like a queue table with id's. My consumer
> thread/s would, poll this table, create a pair and would have to delete
> from this table. But this is a known anti pattern to not use Cassandra as a
> queue causing tombstones etc.
> But I could not think of any other way. Does anyone have any other
> suggestion so as to not delete after a pair is created. Is Cassandra not
> the correct technology for a recon process?
>
> Thanks,
> Sagar
>

--94eb2c0b321a59eb580543f155bc
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Sagar,</div><div>=C2=A0</div><blockquote style=3D"=
margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef=
t:1ex" class=3D"gmail_quote">But this is a known anti pattern to not use Ca=
ssandra as a queue causing tombstones etc.<br>But I could not think of any =
other way. Does anyone have any other suggestion so as to not delete after =
a pair is created</blockquote><div><br></div><div>I believe you could try u=
sing a fixed TTL (defined at the table level for example), then use a TWCS =
compaction strategy and compactions options that would efficiently manage t=
ombstones. A colleague at The Last Pickle just wrote an article about TWCS:=
=C2=A0<a href=3D"http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html">=
http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html</a>, and there is =
a lot more information around, including a talk this year at the summit fro=
m Jeff who contributed with TWCS to Apache Cassandra:=C2=A0<a href=3D"https=
://www.youtube.com/watch?v=3DPWtekUWCIaw">https://www.youtube.com/watch?v=
=3DPWtekUWCIaw</a>.</div><div><br></div><div>Also using a time buckets in t=
he partition key could help making sure tombstones will be correctly remove=
d and are not being scanned when requesting new data. Yet do not use <b>onl=
y</b>=C2=A0a time bucket in the partition key as it would lead to=C2=A0hots=
pots. For a given date, only one node (+ replicas) would handle the write /=
 read load.</div><div><br></div><div>So using &quot;day + something else&qu=
ot; as a partition key and &quot;TWCS + Fixed TTLs&quot; <b>could</b> be a =
good way to move forward.</div><div><br></div><div>I would give it a try wi=
th the cassandra-stress tool that is shipped alongside Apache Cassandra and=
 allows the use of a user defined schema.</div><div><br></div><div>C*heers,=
</div><div><br></div><div>Alain</div></div><div class=3D"gmail_extra"><br><=
div class=3D"gmail_quote">2016-12-17 21:02 GMT+01:00 Sagar Jambhulkar <span=
 dir=3D"ltr">&lt;<a href=3D"mailto:sagar.jambhulkar@gmail.com" target=3D"_b=
lank">sagar.jambhulkar@gmail.com</a>&gt;</span>:<br><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex"><div dir=3D"auto"><div dir=3D"auto" style=3D"font-family:sans-serif=
;font-size:25.4477px">Hi,</div><div dir=3D"auto" style=3D"font-family:sans-=
serif;font-size:25.4477px">Needed a suggestion for a schema query. I want t=
o build a reconciliation using Cassandra. Basically two or more systems sen=
d message to a reconciliation process. The reconciliation process first doe=
s a level one match of id&#39;s and than does complete comparison of messag=
es.</div><div dir=3D"auto" style=3D"font-family:sans-serif;font-size:25.447=
7px"><br></div><div dir=3D"auto" style=3D"font-family:sans-serif;font-size:=
25.4477px">The best I could think of is a like a queue table with id&#39;s.=
 My consumer thread/s would, poll this table, create a pair and would have =
to delete from this table. But this is a known anti pattern to not use Cass=
andra as a queue causing tombstones etc.</div><div dir=3D"auto" style=3D"fo=
nt-family:sans-serif;font-size:25.4477px">But I could not think of any othe=
r way. Does anyone have any other suggestion so as to not delete after a pa=
ir is created. Is Cassandra not the correct technology for a recon process?=
</div><div dir=3D"auto" style=3D"font-family:sans-serif;font-size:25.4477px=
"><br></div><div dir=3D"auto" style=3D"font-family:sans-serif;font-size:25.=
4477px">Thanks,</div><div dir=3D"auto" style=3D"font-family:sans-serif;font=
-size:25.4477px">Sagar</div></div>
</blockquote></div><br></div>

--94eb2c0b321a59eb580543f155bc--