accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billie J Rinaldi <billie.j.rina...@ugov.gov>
Subject Re: Schema Migration (example using Curator)
Date Mon, 19 Mar 2012 18:13:20 GMT
On Friday, March 16, 2012 7:11:19 AM, "David Medinets" <david.medinets@gmail.com> wrote:
> http://www.pgrs.net/2012/03/08/data-migrations-for-nosql-with-curator/
> 
> Would something like this work with Accumulo? Perhaps reserve a column
> family ($SV$ - schema version) name? Maybe a bit of groovy (or other
> dynamic language) can be attached to each table/schema version
> combination so that the database holds all relevant information - even
> the migration code.

In Accumulo you could use an Iterator for data migrations.  The iterator could be applied
at compaction time and scan time so that any clients querying the table would see the new
schema, and the schema change would lazily be written to disk as more data is written to the
table and files are compacted.  In 1.3 it would be possible to tell the table to compact anything
that hadn't been compacted since a certain date (e.g. the date the iterator was applied),
so you could decide at some point to complete the migration and remove the Iterator.  This
is no longer possible in 1.4 because you aren't allowed to request a compaction for a date,
you are only allowed to request a compaction to be started immediately.  If you wanted to
be able to remove the migration iterator, you would have to inspect the tablets manually to
see if they'd been compacted since the migration began and request them to compact if they
hadn't.

Perhaps we could create an ability to mark a table, and an ability to compact anything that
hasn't been compacted since the mark?

Billie

Mime
View raw message