Mailing-List: contact dev-help@samza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.apache.org
MIME-Version: 1.0
In-Reply-To: <D2CED6D1.B6E8%dgarcia@homeaway.com>
References: <D2CED6D1.B6E8%dgarcia@homeaway.com>
From: Jacob Maes <jacob.maes@gmail.com>
Date: Wed, 27 Jan 2016 18:46:25 -0800
Message-ID: 
 <CAHUevGFrOvo+6F8Sq0XJDETtipk3UjxWzOfR0MoE_kuhhq_+1A@mail.gmail.com>
Subject: Re: ChangeLog Question for TTL rocksDB stores
To: dev@samza.apache.org
Content-Type: multipart/alternative; boundary=089e01538360df80a1052a5bedd0

--089e01538360df80a1052a5bedd0
Content-Type: text/plain; charset=UTF-8

Here's my understanding. The others can correct me if I'm mistaken.

Samza provides the changelog functionality by intercepting RocksDB "put"
and "delete" operations. However, TTL is managed by RocksDB internally and
there aren't any hooks exposed in the RocksDB JNI. So there are 2 problems
that arise with TTL and change logging:
1. Samza doesn't know when an entry expires, so it can't delete the expired
entry from the changelog.
2. The changelog currently has no concept of entry age/timestamp, so when
the changelog is restored, it's unknown whether some subset (or all) of the
entries should be immediately expired.

These issues aren't insurmountable, but they weren't pursued for the
initial implementation. Perhaps because there was a shortage of use cases
that needed both TTL and changelogging, but I'm not sure.

-Jake

On Wed, Jan 27, 2016 at 6:19 PM, David Garcia <dgarcia@homeaway.com.invalid>
wrote:

> So, I saw this very scary message:
>
>
> ERROR - e.kv.RocksDbKeyValueStore$ - sessionJoinStore is a TTL based
> store, changelog is not supported for TTL based stores, use at your own
> discretion
>
>
>
>
> A few of questions:
>
> 1.) Does this mean that this store is NOT backed by the changelog?
>
> 2.) Provided that the store IS backed by a change log, do the TTL
> expirations commit removals from the changelog (I.e. Nulls)...presumably
> upon compaction
>
> 3.) Can I please get a bit more detail on how TTL affects a changelog
> store?
>
>
> -David
>

--089e01538360df80a1052a5bedd0--