Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of chris.burroughs@gmail.com
 designates 209.85.216.47 as permitted sender)
Message-ID: <4E2F678E.6060500@gmail.com>
Date: Tue, 26 Jul 2011 21:19:10 -0400
From: Chris Burroughs <chris.burroughs@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.18) Gecko/20110617 Thunderbird/3.1.11
MIME-Version: 1.0
To: kafka-dev@incubator.apache.org
Subject: On time/offset indexs
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

So for good reason [1] Kafka doesn't keep a complicated time --> offset
index.  Whatever is the start and end of log file is what you get.  We
can approximate finer grained time indexes with smaller log files [2]
and getOffsetsBefore, but we would really prefer not to have lots of
small files everywhere.

To solve the case of wanting time based indexes without lots of files
could we have another append only companion file for each Log that
periodically (I'm thinking on the order of 1 minute) gets
timestamp:offset appended to it?  That should have low overhead and if
the companion file is missing/deleted/etc we can still use the current
logic.

[1] "Furthermore the complexity of maintaining the mapping from a random
id to an offset requires a heavy weight index structure which must be
synchronized with disk, essentially requiring a full persistent
random-access data structure. " http://sna-projects.com/kafka/design.php

[2] And KAFKA-40 would make this easier to do.