hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <saint....@gmail.com>
Subject Re: New file format migration
Date Sun, 15 Feb 2009 05:38:29 GMT
Good stuff Md & andrew.  Will make an issue to do an mr as part of  
migration step

On Feb 14, 2009, at 10:57 AM, Andrew Purtell <apurtell@apache.org>  

> Migration as an external tool that uses mapreduce for bulk
> rewrites or updates to file structures seems a better overall
> strategy than in place migration. The code will be cleaner
> everywhere except for the migration tool, which is what I think
> is the most desirable situation.
>   - Andy
>> From: stack
>> Moving to the new file format (see hbase-61), I used to
>> think that we could run the regionserver with readers and
>> writers for both the old and new and that as we went, we'd
>> rewrite old file format into the new on compaction.
> [...]
>> Now, starting the hfile integration effort, I see that lazy
>> migration would force us to give up some of the performance
>> benefits hfile brings.
> [...]
>> I now am tending toward a fat migration that major compacts
>> old stores and as it runs, writes out new files as hfiles.
>> We'd do this as a distinct mapreduce job or add it into
>> regionserver startup -- basically, on open, migrate the
>> individual regions.

View raw message