hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-18640) Move mapreduce out of hbase-server into separate hbase-mapreduce moduel
Date Wed, 23 Aug 2017 04:06:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137845#comment-16137845
] 

stack commented on HBASE-18640:
-------------------------------

Yeah, I ask same up on rb review about hbase-mapreduce-util 

Whats in hbase-mapreduce-util?

I see OutputFormats, Map & Reduce Tasks, Partitioners, Record Readers, a little Util...
Should it be called hbase-mapreduce? Or hbase-mapreduce-core. You have to include this ? 
If so, core? It looks like you might be able to do hbase-related MR work w/ core w/o having
an hbase instance up for some of these jobs -- or not all need to communicate with a running
hbase instance?

Whats in hbase-mapreduce currently?

InputFormats that depend on running being able to get at a running hbase and the MR tests.
 Call it hbase-mapreduce-cluster hbase-mapreduce-runtime (where runtime signifies you need
a running hbase up for most of the included tasks and utils to work) or hbase-mapreduce-client
where client here is client of a running hbase instance?

BTW, there are two Driver classes, one in hbase-mapreduce-util and other in hbase-mapreduce?
One is mapred and other is mapreduce. Can both be in same module?





> Move mapreduce out of hbase-server into separate hbase-mapreduce moduel
> -----------------------------------------------------------------------
>
>                 Key: HBASE-18640
>                 URL: https://issues.apache.org/jira/browse/HBASE-18640
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Appy
>            Assignee: Appy
>         Attachments: HBASE-18640.master.001.patch, HBASE-18640.master.002.patch, HBASE-18640.master.003.patch,
HBASE-18640.master.003.patch, HBASE-18640.master.004.patch, HBASE-18640.master.004.patch,
HBASE-18640.master.005.patch
>
>
> (Couldn't find another dedicated jira, so creating new one).
> Uploaded patch which is moving ~60 files to the new module. Few notes:
> - The classes remaining in hbase-server are the ones which are intensively coupled with
visibility labels/wal/filesystem/hfile. These can not be migrated to new module until corresponding
subcomponents are untangled out of hbase-server into their own separate modules.
> - Almost all mapreduce tests uses HBaseTestingUtil, so they can't be moved to hbase-mapreduce
module. Given these dependency constraints, one way would be having a separate module for
tests:
> hbase-mapreduce <---- hbase-server <------- hbase-mapreduce-tests 
> Imo, this makes sense and looks fine.
> The only issue is - yetus' pre-commit. It won't run tests in hbase-mapreduce-tests module
if something changed in just hbase-mapreduce. However, yetus' limitation shouldn't warrant
against the idea.
> So i'd say that we should go that way, unless there are better suggestions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message