Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of clohfink@blackbirdit.com
 designates 209.85.223.182 as permitted sender)
From: Chris Lohfink <clohfink@blackbirdit.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_708E800D-573D-4DCE-8CDC-6C6446488B41"
Message-Id: <C65D0834-8665-4A38-990F-A4586BF4459B@blackbirdit.com>
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
Subject: Re: Storing log structured data in Cassandra without compactions for
 performance boost.
Date: Wed, 7 May 2014 15:20:02 -0500
References: 
 <CAAZU44mjUSce_1ta0Zyvk5nyYxbo=vvOkbcaQmxV6kDZLabCFA@mail.gmail.com>
To: user@cassandra.apache.org
In-Reply-To: 
 <CAAZU44mjUSce_1ta0Zyvk5nyYxbo=vvOkbcaQmxV6kDZLabCFA@mail.gmail.com>


--Apple-Mail=_708E800D-573D-4DCE-8CDC-6C6446488B41
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

Whats your data model look like?

> I think it would be best to just disable compactions.

Why? are you never doing reads?  There is also a cost to =
repairs/bootstrapping when you have a ton of sstables.  This might be a =
premature optimization.

If the data is read from a slice of a partition that has been added over =
time there will be a part of that row in every almost sstable. That =
would mean all of them (multiple disk seeks depending on clustering =
order per sstable) would have to be read from in order to service the =
query.  Data model can help or hurt a lot though.

If you set the TTL for the columns you added then C* will clean up =
sstables (if size tiered and post 1.2) once the datas been expired.  =
Since you never delete set the gc_grace_seconds to 0 so the ttl =
expiration doesnt result in tombstones.

---
Chris Lohfink=20


On May 6, 2014, at 7:55 PM, Kevin Burton <burton@spinn3r.com> wrote:

> I'm looking at storing log data in Cassandra=85=20
>=20
> Every record is a unique timestamp for the key, and then the log line =
for the value.
>=20
> I think it would be best to just disable compactions.
>=20
> - there will never be any deletes.
>=20
> - all the data will be accessed in time range (probably partitioned =
randomly) and sequentially.
>=20
> So every time a memtable flushes, we will just keep that SSTable =
forever. =20
>=20
> Compacting the data is kind of redundant in this situation.
>=20
> I was thinking the best strategy is to use setcompactionthreshold and =
set the value VERY high to compactions are never triggered.
>=20
> Also, It would be IDEAL to be able to tell cassandra to just drop a =
full SSTable so that I can truncate older data without having to do a =
major compaction and without having to mark everything with a tombstone. =
 Is this possible?
>=20
>=20
>=20
> --=20
>=20
> Founder/CEO Spinn3r.com
> Location: San Francisco, CA
> Skype: burtonator
> blog: http://burtonator.wordpress.com
> =85 or check out my Google+ profile
>=20
> War is peace. Freedom is slavery. Ignorance is strength. Corporations =
are people.
>=20


--Apple-Mail=_708E800D-573D-4DCE-8CDC-6C6446488B41
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=windows-1252

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space;"><div>Whats your data model look =
like?</div><br><blockquote type=3D"cite"><div dir=3D"ltr">I think it =
would be best to just disable =
compactions.</div></blockquote><div><br></div>Why? are you never doing =
reads? &nbsp;There is also a cost to repairs/bootstrapping when you have =
a ton of sstables. &nbsp;This might be a premature =
optimization.<div><br></div><div>If the data is read from a slice of a =
partition that has been added over time there will be a part of that row =
in every almost sstable. That would mean all of them (multiple disk =
seeks depending on clustering order per sstable) would have to be read =
from in order to service the query. &nbsp;Data model can help or hurt a =
lot though.</div><div><br></div><div>If you set the TTL for the columns =
you added then C* will clean up sstables (if size tiered and post 1.2) =
once the datas been expired. &nbsp;Since you never delete set the =
gc_grace_seconds to 0 so the ttl expiration doesnt result in =
tombstones.</div><div><br></div><div>---</div><div>Chris =
Lohfink&nbsp;<br><div><br></div><div><br><div><br><div><div>On May 6, =
2014, at 7:55 PM, Kevin Burton &lt;<a =
href=3D"mailto:burton@spinn3r.com">burton@spinn3r.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div dir=3D"ltr">I'm looking at storing log data in =
Cassandra=85&nbsp;<div><br></div><div>Every record is a unique timestamp =
for the key, and then the log line for the =
value.</div><div><br></div><div>I think it would be best to just disable =
compactions.</div>

<div><br></div><div>- there will never be any =
deletes.</div><div><br></div><div>- all the data will be accessed in =
time range (probably partitioned randomly) and =
sequentially.</div><div><br></div><div>So every time a memtable flushes, =
we will just keep that SSTable forever. &nbsp;</div>

<div><br></div><div>Compacting the data is kind of redundant in this =
situation.</div><div><br></div><div>I was thinking the best strategy is =
to use setcompactionthreshold and set the value VERY high to compactions =
are never triggered.</div>

<div><br></div><div>Also, It would be IDEAL to be able to tell cassandra =
to just drop a full SSTable so that I can truncate older data without =
having to do a major compaction and without having to mark everything =
with a tombstone. &nbsp;Is this possible?</div>

<div><br></div><div><br clear=3D"all"><div><br></div>-- <br><div><div =
style=3D"margin: 0px 0px 12pt;"><br =
class=3D"webkit-block-placeholder"></div><div>Founder/CEO&nbsp;<a =
href=3D"http://spinn3r.com/" target=3D"_blank">Spinn3r.com</a><br>

</div><div>Location:&nbsp;<b>San Francisco, =
CA</b><br>Skype:&nbsp;<b>burtonator</b></div><div><font color=3D"#2c2c2c" =
face=3D"Helvetica, Arial, sans-serif"><span =
style=3D"line-height:19px">blog:<b>&nbsp;</b></span></font><a =
href=3D"http://burtonator.wordpress.com/" =
target=3D"_blank">http://burtonator.wordpress.com</a></div>

<div>=85 or check out my <a =
href=3D"https://plus.google.com/102718274791889610666/posts" =
target=3D"_blank">Google+ profile</a></div><div><a =
href=3D"http://spinn3r.com/" target=3D"_blank"><img =
src=3D"http://spinn3r.com/images/spinn3r.jpg"></a></div>

<div><span style=3D"font-family: verdana, arial, helvetica, sans-serif; =
font-size: small; font-style: normal; font-variant: normal; font-weight: =
normal; letter-spacing: normal; line-height: normal; text-align: start; =
text-indent: 0px; text-transform: none; white-space: normal; =
word-spacing: 0px; background-color: rgb(255, 255, 255); float: none; =
display: inline !important;">War is peace. Freedom is slavery. Ignorance =
is strength. Corporations are people.</span></div><div><br =
class=3D"webkit-block-placeholder"></div></div>
</div></div>
</blockquote></div><br></div></div></div></body></html>=

--Apple-Mail=_708E800D-573D-4DCE-8CDC-6C6446488B41--