hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wymer (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5140) TableInputFormat subclass to allow N number of splits per region during MR jobs
Date Fri, 06 Jan 2012 20:12:39 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13181565#comment-13181565
] 

Josh Wymer commented on HBASE-5140:
-----------------------------------

One glaring issue is the lack of start & end keys for one region tables. To get the start
key we could do a quick scan of the first row and get the key. For the last region of a table,
I'm not sure how we'll handle determining the end key other than setting it to the max size
of whatever data type (e.g. long) we are using for the split calculations. Any suggestions
other than this?
                
> TableInputFormat subclass to allow N number of splits per region during MR jobs
> -------------------------------------------------------------------------------
>
>                 Key: HBASE-5140
>                 URL: https://issues.apache.org/jira/browse/HBASE-5140
>             Project: HBase
>          Issue Type: New Feature
>          Components: mapreduce
>            Reporter: Josh Wymer
>            Priority: Trivial
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> In regards to [HBASE-5138|https://issues.apache.org/jira/browse/HBASE-5138] I am working
on a subclass for the TableInputFormat class that overrides getSplits in order to generate
N number of splits per regions and/or N number of splits per job. The idea is to convert the
startKey and endKey for each region from byte[] to BigDecimal, take the difference, divide
by N, convert back to byte[] and generate splits on the resulting values. Assuming your keys
are fully distributed this should generate splits at nearly the same number of rows per split.
Any suggestions on this issue are welcome.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message