hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From llpind <sonny_h...@hotmail.com>
Subject Re: Help with Map/Reduce program
Date Thu, 11 Jun 2009 05:15:31 GMT

Sorry I forgot to mention the overflow then overflows into new row keys per
10,000 column entries (or some other split number).



llpind wrote:
> 
> 
> When is the plan for releasing .20?  This particular issue is really
> important to us. 
> 
> Stack, I also have another question: The problem we are trying to solve
> doesn't really need the extra layer present in HBase (BigTable) structure
> (RowResult holds row key and a HashMap of column name, value). What we
> really need is a row key which simply holds a set of values.  Essentially
> this is a many-to-many.  I wanted your thoughts on how we can go about
> solving this problem (we can start another post for this if you’d like).
> Is this something HBase can solve, or something that could potentially be
> a HBase fork?  Right now we are still in test mode, and only having to
> deal with millions of columns, but in production (if the company sticks
> with HBase) the columns could be in the billions.  One idea we came up
> with is to have an overflow table… e.g.
> 
> For a given row key we list the first 10,000 columns (values in our case),
> and after that we create a column with an overflow id pointing an overflow
> table which is keyed on this id.
> 
> This appears it may work, but isn’t the most elegant solution.  I’d
> appreciate input from anyone on this issue.   Please, let me know if you
> need me explain our problem in more detail. 
> 
> 
> 
> stack-3 wrote:
>> 
>> On Wed, Jun 10, 2009 at 4:52 PM, llpind <sonny_heer@hotmail.com> wrote:
>> 
>>>
>>> Thanks.  I think the problem is I have potentially millions of columns.
>>>
>> 
>>> where a given RowResult can hold millions of columns to values.   Thats
>>> why
>>> Map/Reduce is having problems as well (Java Heap exception).  I've upped
>>> mapred.child.java.opts, but problem presists.
>>>
>> 
>> See also HBASE-867: https://issues.apache.org/jira/browse/HBASE-867
>> St.Ack
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Help-with-Map-Reduce-program-tp23952252p23975432.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message