hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: New file format migration
Date Sat, 14 Feb 2009 15:33:28 GMT
As long as this procedure is extensively tested and covered in
documentation, +1. We could call for testers in the user list in the hope of
getting 2-3 non-dev to test it and get feedback once it works on our end.

Good stuff guys.

J-D

On Fri, Feb 13, 2009 at 6:13 PM, stack <stack@duboce.net> wrote:

> Moving to the new file format (see hbase-61), I used to think that we could
> run the regionserver with readers and writers for both the old and new and
> that as we went, we'd rewrite old file format into the new on compaction.
> To facilitate the lazy migration, we would spend time extracting a
> lowest-common denominator interface and then write implementations and glue
> code for mapfile and the new hfile.
>
> Now, starting the hfile integration effort, I see that lazy migration would
> force us to give up some of the performance benefits hfile brings.  Whole
> sections of the Store and StoreFile code are synchronized to allow the
> mapfile iterate undisturbed without the inteference of any another
> concurrent access.  This blocking is not necessary with hfile (internally
> it
> maintains thread-safe scanning).  Writing two code paths, one for the old
> format and one for the new would complicate the server considerably, delay
> the release, etc.
>
> I now am tending toward a fat migration that major compacts old stores and
> as it runs, writes out new files as hfiles.  We'd do this as a distinct
> mapreduce job or add it into regionserver startup -- basically, on open,
> migrate the individual regions.
>
> Comments?
> Thanks,
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message