Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates
 209.85.217.227 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=KBbVLu4uQgxjdswLPuMC9Ffj+5KXtw63gh+3cmhx7ihuP0yZh2Q2jvT0Y5WkG1Gqrj
         oBNRsX6nN/rhP1x2/AkXRChwQIu4l1siquZ8Q+csgS7QwWLu7wwwYsYc0k3goVOarYEb
         YhEiEvhILLLzJ1cr9OmFlxk6VHRDP5gku2134=
MIME-Version: 1.0
In-Reply-To: <7c962aed0907271352w65fb999gda863ca4b5c8d001@mail.gmail.com>
References: <32120a6a0907271331i73b63f3fiddcfe26928d18fd8@mail.gmail.com>
	 <7c962aed0907271352w65fb999gda863ca4b5c8d001@mail.gmail.com>
Date: Mon, 27 Jul 2009 13:56:40 -0700
Message-ID: <78568af10907271356k66c32a53web3a164176d1c4e2@mail.gmail.com>
Subject: Re: Fast importing into HBase (bypassing RegionServer)
From: Ryan Rawson <ryanobjc@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The last time I seriously looked at this, it was to answer serious
performance issues with HBase.  I eventually fixed said performance
issues, and thus went on to drop the idea overall.

-ryan

On Mon, Jul 27, 2009 at 1:52 PM, stack<stack@duboce.net> wrote:
> Latest thinking is write a MR job that in the reducer writes hfiles that =
are
> just under a region size (<256M). =A0When reducer has reached about 240MB=
, it
> opens new file. =A0(May need to write custom ReduceRunner to keep account=
 of
> whats been written and to rotate the file).
>
> After the MR has finished, a script would come along, move the hfiles int=
o
> appropriate directory structure. =A0Each hfile would be the sole content =
of
> the region. =A0The script would read from each hfile's metadata its first=
 and
> last keys and then using this metainfo along with a table format specifie=
d
> externally, insert an entry into .META. per region (See the scripts in bi=
n
> -- copy and rename table -- for examples of how to manipulate .META.).
>
> Someone needs to just do it. =A0We've been talking about it for ever.
>
> St.Ack
> P.S. Here is older thinking on the topic
> https://issues.apache.org/jira/browse/HBASE-48
>
> On Mon, Jul 27, 2009 at 1:31 PM, tim robertson <timrobertson100@gmail.com=
>wrote:
>
>> Hi all,
>>
>> Ryan wrote on a different thread:
>>
>> "It should be possible to randomly insert data from a pre-existing
>> data set. =A0There is some work to directly import straight into hfiles
>> and skipping the regionserver, but that would only really work on 1
>> time imports to new tables."
>>
>> Could someone please elaborate on this a little and outline the steps
>> needed? =A0Do you write an hfile in a custom mapreduce output format and
>> then somehow write the table metadata file afterwards?
>>
>> Cheers,
>>
>> Tim
>>
>