Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 74.125.92.25 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=jRNVGbgbkiqVwg/GWk/BGJ7DewpyyAwBO+BKbmF7eZsnj2A2yk3YqTikE0f41KNbOW
         6jtXCzSq1TY5zbIlQVpt7s535D3Gl2Wllqu22sGlTZ1bJ8G+iwO12nLILgtAZvuIwwT8
         0sq//Xrr6zBy3TMc/L9fWMTA+DECfUa4jPvgs=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <32120a6a0907271331i73b63f3fiddcfe26928d18fd8@mail.gmail.com>
References: <32120a6a0907271331i73b63f3fiddcfe26928d18fd8@mail.gmail.com>
Date: Mon, 27 Jul 2009 13:52:06 -0700
Message-ID: <7c962aed0907271352w65fb999gda863ca4b5c8d001@mail.gmail.com>
Subject: Re: Fast importing into HBase (bypassing RegionServer)
From: stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163642749e7c5dc7046fb6200f

--00163642749e7c5dc7046fb6200f
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Latest thinking is write a MR job that in the reducer writes hfiles that are
just under a region size (<256M).  When reducer has reached about 240MB, it
opens new file.  (May need to write custom ReduceRunner to keep account of
whats been written and to rotate the file).

After the MR has finished, a script would come along, move the hfiles into
appropriate directory structure.  Each hfile would be the sole content of
the region.  The script would read from each hfile's metadata its first and
last keys and then using this metainfo along with a table format specified
externally, insert an entry into .META. per region (See the scripts in bin
-- copy and rename table -- for examples of how to manipulate .META.).

Someone needs to just do it.  We've been talking about it for ever.

St.Ack
P.S. Here is older thinking on the topic
https://issues.apache.org/jira/browse/HBASE-48

On Mon, Jul 27, 2009 at 1:31 PM, tim robertson <timrobertson100@gmail.com>wrote:

> Hi all,
>
> Ryan wrote on a different thread:
>
> "It should be possible to randomly insert data from a pre-existing
> data set.  There is some work to directly import straight into hfiles
> and skipping the regionserver, but that would only really work on 1
> time imports to new tables."
>
> Could someone please elaborate on this a little and outline the steps
> needed?  Do you write an hfile in a custom mapreduce output format and
> then somehow write the table metadata file afterwards?
>
> Cheers,
>
> Tim
>

--00163642749e7c5dc7046fb6200f--