chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiaqi Tan <>
Subject Re: Backfilling and retroactive table creation in MySQL database
Date Thu, 11 Jun 2009 23:02:48 GMT

Answering my own question, I had a dirty hack to just replace the date
argument passed to the TableCreator and specify the date at which the
data-to-be-inserted was timestamped so that the right tables into the
future get created.

Two issues: 1. I'm not sure if that's the cleanest solution, and 2.
perhaps this should be dealt with in conjunction with the backfilling
work, especially if the backfilling process is aware of the timestamp
of the data being backfilled into the Chukwa storage.


On Thu, Jun 11, 2009 at 3:59 PM, Ariel Rabkin<> wrote:
> I'd be fine with doing retroactive table creation.  Though I'd sort of
> like to see us move towards a cleaner database architecture -- the
> whole system of having template tables, and then a whole series of
> tables for time partitions always struck me as sort of awkward.
> But I'm not really a database guy, so I don't know what the Right Thing is. :(
> On Thu, Jun 11, 2009 at 3:42 PM, Jiaqi Tan<> wrote:
>> Hi,
>> I've been trying to use the MDL to retroactively load data that was
>> generated at an earlier time (somewhat like backfilling, but not,
>> because I'm loading my own state-machine data), but when I create new
>> tables and fire up in the present time, it only creates
>> tables into the future. Is there any way to create tables for time
>> periods that have passed?
>> Also, taking a step back, if we have support for backfilling, should
>> there also be support for retroactive table creation for the database,
>> for instance, for bulk-loading trace data that wasn't collected using
>> the Agent+Collector framework?
>> Jiaqi
> --
> Ari Rabkin
> UC Berkeley Computer Science Department

View raw message