Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: Apache Wiki <wikidiffs@apache.org>
To: Apache Wiki <wikidiffs@apache.org>
Date: Fri, 09 Apr 2010 00:58:51 -0000
Message-ID: <20100409005851.19112.56730@eos.apache.org>
Subject: 
 =?utf-8?q?=5BHadoop_Wiki=5D_Update_of_=22Hive/HBaseBulkLoad=22_by_JohnSic?=
 =?utf-8?q?hi?=

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for ch=
ange notification.

The "Hive/HBaseBulkLoad" page has been changed by JohnSichi.
http://wiki.apache.org/hadoop/Hive/HBaseBulkLoad?action=3Ddiff&rev1=3D1&rev=
2=3D2

--------------------------------------------------

  =

  =3D Overview =3D
  =

- Ideally, bulk load from Hive into HBase would be as simple as this:
+ Ideally, bulk load from Hive into HBase would be part of [[Hive/HBaseInte=
gration]], making it as simple as this:
  =

  {{{
  CREATE TABLE new_hbase_table(rowkey string, x int, y int) =

@@ -19, +19 @@

  SELECT ... FROM hive_query;
  }}}
  =

- However, things aren't ''quite'' as simple as that yet.  Instead, a multi=
step procedure is required involving both SQL and shell script commands.  I=
t should still be a lot easier and more flexible than writing your own map/=
reduce program, and over time we can enhance Hive to move closer to the ide=
al.
+ However, things aren't ''quite'' as straightforward as that yet.  Instead=
, a procedure involving a series of SQL commands is required.  It should st=
ill be a lot easier and more flexible than writing your own map/reduce prog=
ram, and over time we hope to enhance Hive to move closer to the ideal.
  =

  The procedure is based on [[http://hadoop.apache.org/hbase/docs/r0.20.2/a=
pi/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk|underlying H=
Base recommendations]], and involves the following steps:
  =

   1. Decide on the number of reducers you're planning to use for paralleli=
zing the sorting and HFile creation.  This depends on the size of your data=
 as well as cluster resources available.
-  1. Run Hive commands which will create a file containing "splitter" keys=
 which will be used for range-partitioning the data during sort.
+  1. Run Hive sampling commands which will create a file containing "split=
ter" keys which will be used for range-partitioning the data during sort.
   1. Prepare a staging location in HDFS where the HFiles will be generated.
   1. Run Hive commands which will execute the sort and generate the HFiles.
   1. (Optional:  if HBase and Hive are running in different clusters, dist=
cp the generated files from the Hive cluster to the HBase cluster.)
@@ -33, +33 @@

  =

  The rest of this page explains each step in greater detail.
  =

+ =3D Estimate Resources Needed =3D
+ =

+ '''tbd:  provide some example numbers based on Facebook experiments'''
+ =

+ =3D Run Sampling for Range Partitioning =3D
+ =

+ =3D Prepare Staging Location =3D
+ =

+ =3D Sort Data =3D
+ =

+ =3D Run HBase Script =3D
+=20