Mailing-List: contact dev-help@samza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.apache.org
From: David Garcia <dgarcia@homeaway.com.INVALID>
To: "dev@samza.apache.org" <dev@samza.apache.org>
Subject: Re: ChangeLog Question for TTL rocksDB stores
Thread-Topic: ChangeLog Question for TTL rocksDB stores
Thread-Index: AQHRWXJR/XJFw5n5J0meARQQ4omxAJ8QOUOAgAC1lgA=
Date: Thu, 28 Jan 2016 19:36:21 +0000
Message-ID: <D2CFC92F.B76C%dgarcia@homeaway.com>
References: <D2CED6D1.B6E8%dgarcia@homeaway.com>
 <CAHUevGFrOvo+6F8Sq0XJDETtipk3UjxWzOfR0MoE_kuhhq_+1A@mail.gmail.com>
In-Reply-To: 
 <CAHUevGFrOvo+6F8Sq0XJDETtipk3UjxWzOfR0MoE_kuhhq_+1A@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [64.245.52.2]
x-microsoft-exchange-diagnostics: 
 1;BY2PR08MB015;5:8uXtSzfNA5wOe5bKpVJM476qzeISyaikZ7xYjvHwANOMhwxQfaO93LKowLYlSe3pQiBLNP798KkTvxvRZvBR2iNz0UKlDMscSGSP/fIz0/S3P7w6fLoEl2oGjOvJK/mMN8xQqIa+nK7fiKy34p8Ezw==;24:douUgdm3+AeB8nsMkWo59/uSOd94A0T/UGoDQ6Lc/vMxgEDChGPzwDI8tH7zgKv3msqhVFBTIDTRK+xy4aq6QuQGnirUINlhXgo1LKh7f/o=
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR08MB015;
x-ms-office365-filtering-correlation-id: fc6d9529-d215-4fcb-960e-08d3281a4d8c
x-microsoft-antispam-prvs: 
 <BY2PR08MB015614BA34FDE3F4F5D0023CFDA0@BY2PR08MB015.namprd08.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:;
x-exchange-antispam-report-cfa-test: 
 BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001);SRVR:BY2PR08MB015;BCL:0;PCL:0;RULEID:;SRVR:BY2PR08MB015;
x-forefront-prvs: 083526BF8A
x-forefront-antispam-report: 
 SFV:NSPM;SFS:(10009020)(6009001)(479174004)(24454002)(377454003)(189002)(199003)(54534003)(5001960100002)(3846002)(102836003)(54356999)(2950100001)(1096002)(97736004)(107886002)(2900100001)(11100500001)(2351001)(19580395003)(110136002)(586003)(106116001)(3470700001)(106356001)(101416001)(50986999)(5002640100001)(105586002)(189998001)(66066001)(99286002)(76176999)(1220700001)(3280700002)(5008740100001)(2906002)(87936001)(122556002)(19580405001)(6116002)(10400500002)(5004730100002)(81156007)(10090500001)(86362001)(36756003)(450100001)(92566002)(40100003)(2501003)(3660700001)(59356010)(207903002)(219803003)(94096001);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR08MB015;H:BY2PR08MB013.namprd08.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en;
received-spf: None (protection.outlook.com: homeaway.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <A0C6D466588ECD4C8A19D3E1887361B5@namprd08.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jan 2016 19:36:21.7515
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: e3833796-b508-4841-83ae-38f7823ef2ae
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR08MB015
X-OrganizationHeadersPreserved: BY2PR08MB015.namprd08.prod.outlook.com
X-CrossPremisesHeadersPromoted: ascmail004.wvrgroup.internal
X-CrossPremisesHeadersFiltered: ascmail004.wvrgroup.internal
X-OriginatorOrg: homeaway.com

Ok, that makes sense.  I had assumed that the changelog was supported
because the docs mention that TTL is enforced upon =B3compaction=B2 (I had
assumed compaction of the DB changelog).  Which topic does the TTL policy
listen for the compaction of (since compaction policies of topics can
differ)?

-David

On 1/27/16, 8:46 PM, "Jacob Maes" <jacob.maes@gmail.com> wrote:

>Here's my understanding. The others can correct me if I'm mistaken.
>
>Samza provides the changelog functionality by intercepting RocksDB "put"
>and "delete" operations. However, TTL is managed by RocksDB internally and
>there aren't any hooks exposed in the RocksDB JNI. So there are 2 problems
>that arise with TTL and change logging:
>1. Samza doesn't know when an entry expires, so it can't delete the
>expired
>entry from the changelog.
>2. The changelog currently has no concept of entry age/timestamp, so when
>the changelog is restored, it's unknown whether some subset (or all) of
>the
>entries should be immediately expired.
>
>These issues aren't insurmountable, but they weren't pursued for the
>initial implementation. Perhaps because there was a shortage of use cases
>that needed both TTL and changelogging, but I'm not sure.
>
>-Jake
>
>On Wed, Jan 27, 2016 at 6:19 PM, David Garcia
><dgarcia@homeaway.com.invalid>
>wrote:
>
>> So, I saw this very scary message:
>>
>>
>> ERROR - e.kv.RocksDbKeyValueStore$ - sessionJoinStore is a TTL based
>> store, changelog is not supported for TTL based stores, use at your own
>> discretion
>>
>>
>>
>>
>> A few of questions:
>>
>> 1.) Does this mean that this store is NOT backed by the changelog?
>>
>> 2.) Provided that the store IS backed by a change log, do the TTL
>> expirations commit removals from the changelog (I.e. Nulls)...presumably
>> upon compaction
>>
>> 3.) Can I please get a bit more detail on how TTL affects a changelog
>> store?
>>
>>
>> -David
>>