Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of clohfink@blackbirdit.com
 designates 209.85.213.178 as permitted sender)
From: Chris Lohfink <clohfink@blackbirdit.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_FB457941-B252-4722-9126-48D5E25C1A51"
Message-Id: <D4135463-DFB6-487E-8742-857F81F5F596@blackbirdit.com>
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
Subject: Re: do Cassandra generate a event or log containing key value of
 column when a column expires due to TTL
Date: Fri, 22 Aug 2014 09:24:22 -0500
References: 
 <CAFSAEXTDTcx2NPc070K0dHRrn_XF-SBCefsL2iJ+Kv5kS9N2Dw@mail.gmail.com>
To: user@cassandra.apache.org
In-Reply-To: 
 <CAFSAEXTDTcx2NPc070K0dHRrn_XF-SBCefsL2iJ+Kv5kS9N2Dw@mail.gmail.com>


--Apple-Mail=_FB457941-B252-4722-9126-48D5E25C1A51
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Few options I can think of, probably some better ideas out there.  These =
mostly depending on size of data and how frequently updated.

1) a map reduce or spark job to filter out non-empty rows
2) add some logging and do a custom build of cassandra (ie in =
"removeDeletedCF" of ColumnFamilyStore) and grep log files to get a =
report that can be consumed by your system.  Potentially lotta redundant =
data and may be long after actual expiration before shows up.
3) when you insert your TTLed columns you also write to a wide row, with =
timestamp of expiration and use that to drive the report... there will =
be a bit to that though and a MR job still might be good idea for doing =
processing.

---
Chris

On Aug 22, 2014, at 6:02 AM, Gaurav Bhatnagar <gbhatnagar@gmail.com> =
wrote:

> Hi,
>     I have stored following data structure in cassandra
>=20
> RowKey: 119551747098
>=20
>=20
> =3D> (name=3Dc:per:@batchId, value=3Dad1, timestamp=3D1408345109805011, =
ttl=3D1436489)
>=20
> =3D> (name=3Dc:per:@currency, value=3DUSD, timestamp=3D1408345109805009,=
 ttl=3D1436489)
>=20
> =3D> (name=3Dc:per:@decimalValue, value=3D2, =
timestamp=3D1408345109805003, ttl=3D1436489)
>=20
>=20
>=20
> here Rowkey 119551747098 is a numeric number containing serial number =
of data
>=20
>=20
>=20
> These columns get expired when ttl value for that column is reached.
>=20
> I what to generate an audit trail which contains value of RowKey along =
with column name and value when they get deleted due to ttl expiration.
>=20
>=20
>=20
> I want this audit trail for reconciliation purpose so that I can know =
which all RowKeys have got deleted from the system.
>=20
>=20
>=20
> Is there any way in cassandra through which I can print value of =
RowKeys which get deleted due TTL expiration.
>=20
>=20
>=20
> Regards,
>=20
> Gaurav
>=20


--Apple-Mail=_FB457941-B252-4722-9126-48D5E25C1A51
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Few =
options I can think of, probably some better ideas out there. =
&nbsp;These mostly depending on size of data and how frequently =
updated.<div><br></div><div>1) a map reduce or spark job to filter out =
non-empty rows</div><div>2) add some logging and do a custom build of =
cassandra (ie in "removeDeletedCF" of ColumnFamilyStore) and grep log =
files to get a report that can be consumed by your system. =
&nbsp;Potentially lotta redundant data and may be long after actual =
expiration before shows up.</div><div>3) when you insert your TTLed =
columns you also write to a wide row, with timestamp of expiration and =
use that to drive the report... there will be a bit to that though and a =
MR job still might be good idea for doing =
processing.</div><div><br></div><div>---</div><div>Chris</div><div><br></d=
iv><div><div><div>On Aug 22, 2014, at 6:02 AM, Gaurav Bhatnagar &lt;<a =
href=3D"mailto:gbhatnagar@gmail.com">gbhatnagar@gmail.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div dir=3D"ltr"><div><div>Hi,<br></div>&nbsp;&nbsp;&nbsp; =
I have stored following data structure in cassandra<br><br></div><p =
class=3D"MsoNormal">RowKey: 119551747098</p><div><br =
class=3D"webkit-block-placeholder"></div><p class=3D"MsoNormal">=3D&gt; =
(name=3Dc:per:@batchId, value=3Dad1,
timestamp=3D1408345109805011, ttl=3D1436489)</p><p =
class=3D"MsoNormal">=3D&gt; (name=3Dc:per:@currency, value=3DUSD, =
timestamp=3D1408345109805009,
ttl=3D1436489)</p><p class=3D"MsoNormal">=3D&gt; =
(name=3Dc:per:@decimalValue, value=3D2,
timestamp=3D1408345109805003, ttl=3D1436489)</p><p =
class=3D"MsoNormal"><br></p><p class=3D"MsoNormal">here Rowkey =
119551747098 is a numeric number containing serial number of data</p><p =
class=3D"MsoNormal"><br></p><p class=3D"MsoNormal">
These columns get expired when ttl value for that column is =
reached.</p><p class=3D"MsoNormal">I what to generate an audit trail =
which contains value of RowKey along with column name and value when =
they get deleted due to ttl expiration.</p><p =
class=3D"MsoNormal"><br></p><p class=3D"MsoNormal">I want this audit =
trail for reconciliation purpose so that I can know which all RowKeys =
have got deleted from the system.</p><p class=3D"MsoNormal"><br></p><p =
class=3D"MsoNormal">
Is there any way in cassandra through which I can print value of RowKeys =
which get deleted due TTL expiration.</p><p class=3D"MsoNormal"><br></p><p=
 class=3D"MsoNormal">Regards,</p><p class=3D"MsoNormal">Gaurav<br></p>

</div>
</blockquote></div><br></div></body></html>=

--Apple-Mail=_FB457941-B252-4722-9126-48D5E25C1A51--