hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Sichi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HIVE-1295) facilitate HBase bulk loads from Hive
Date Mon, 12 Apr 2010 21:26:50 GMT

     [ https://issues.apache.org/jira/browse/HIVE-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

John Sichi updated HIVE-1295:

    Status: Patch Available  (was: Open)

Notes on the patch:

* After the recent Apache JIRA downtime, the ASF copyright license assignment radio button
is no longer here in the UI, so I couldn't click that to grant to Apache, but I hereby grant
it etc etc (and I'm a Facebook employee anyway, so I think my contribution is covered under
the corporate license assignment).

* The unit test requires MiniMR mode (instead of local mode) because it needs multiple reducers
in order for the custom partitioner to kick in.  Accordingly, I named the file with a .m suffix
(instead of .q) to keep it segregated from the other unit tests.  To run it, use -Dtestcase=TestHBaseMinimrCliDriver.

* I had to make one change to the classpath definition in build-common.xml to avoid a webapp
ordering issue from HBase and Hadoop being loaded in the same Jetty instance.

* I made one change to ql/build.xml's classpath (adding jsp-2.1 jars).  This was not actually
needed for my hbase-handler test to run, but it fixes an existing breakage with running ql
tests with -Dclustermode=miniMR, needed for HIVE-117.  (The job tracker wasn't starting because
it wasn't able to load some JSP API classes.)


> facilitate HBase bulk loads from Hive
> -------------------------------------
>                 Key: HIVE-1295
>                 URL: https://issues.apache.org/jira/browse/HIVE-1295
>             Project: Hadoop Hive
>          Issue Type: Improvement
>          Components: HBase Handler
>    Affects Versions: 0.6.0
>            Reporter: John Sichi
>            Assignee: John Sichi
>             Fix For: 0.6.0
>         Attachments: HIVE-1295.1.patch
> HBase supports a bulk load procedure:
> http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulk
> We would like to add support to Hive so that users can bulk load HBase from Hive without
having to write any map/reduce code.
> Ideally, this could be done with a single INSERT statement targeting the HBase storage
handler (with an option set to request bulk load instead of row-level inserts).
> However, that will take a lot of work, so this JIRA is a first step to allow the bulk
load files to be prepared inside of Hive via a sequence of SQL statements and then pushed
into HBase via the loadtable.rb script.
> Note that until HBASE-1861 is implemented, the bulk load target table can only have a
single column family.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message