hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dru Jensen <drujen...@gmail.com>
Subject Re: Accessing rows with number indexes
Date Sun, 11 Jan 2009 00:01:28 GMT
I'm not sure this will work or a good idea but is it possible to use  
the tableindexed feature in 0.19 and create an IndexKeyGenerator that  
does an auto increment?

http://svn.apache.org/viewvc/hadoop/hbase/trunk/src/java/org/apache/hadoop/hbase/client/tableindexed/package.html?view=markup


On Jan 10, 2009, at 10:32 AM, Jim Twensky wrote:

> Unfortunately, yes the sentences need to be sorted. I take advantage  
> of the
> lexicographical ordering of the sentences for some other purpose.  
> Even if I
> didn't, how could I generate the prefixes? Do you mean number prefixes
> should be in the range [1-n] where n is the number of rows in the  
> table?
> Since I use Hadoop to pull the data in, I can't see a trivial way to
> generate number prefixes but I may be missing something obvious.
>
> Jim
>
> On Sat, Jan 10, 2009 at 11:55 AM, Tim Sell <trsell@gmail.com> wrote:
>
>> Do the sentences need to be sorted?
>> if not you could use an number prefix on the row key. Keep track of
>> the highest prefix and use that range to select a prefix randomly.
>> Then start a scanner at that prefix
>>
>> ~Tim.
>>
>> 2009/1/10 Jim Twensky <jim.twensky@gmail.com>:
>>> Hello,
>>>
>>> I have an HBase table that contains sentences as row keys and a few
>> numeric
>>> values as columns. A simple abstract model of the table looks like  
>>> the
>>> following:
>>>
>>>
>> --------------------------------------------------------------------------------------------------------------------------
>>> Sentence     |          frequency:value     |       
>>> probability:value-0
>>> |     probability:value-2
>>>
>> --------------------------------------------------------------------------------------------------------------------------
>>> Hello World |                 5                    |       
>>> 0.000545321
>>> |     0.002368204
>>>    .                              .
>>> .                             .
>>>    .                              .
>>> .                             .
>>>    .                              .
>>> .                             .
>>>
>> --------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> I create the table and load it using Hadoop and there are hundreds  
>>> of
>>> billions of entries in it. I use this table to solve an optimization
>> problem
>>> using a hill climbing/simulated annealing method. Basically, I  
>>> need to
>>> change the likelihood values randomly. For example, I need to  
>>> change say
>> the
>>> first 5 rows starting at the 112th row and do some calculations  
>>> and so
>> on...
>>>
>>> Now the problem is, I can't see an easy way to access to the n'th  
>>> row
>>> directly. If I was using a traditional RDBMS, I'd add another  
>>> column and
>>> auto-increment it each time I added a new row but this is not  
>>> possible
>> since
>>> I load the table using Hadoop and the there are parallel insertions
>> taking
>>> place simultaneously. A quick and dirty way to do this might be  
>>> adding a
>> new
>>> index column after I load and initialize the table but the table  
>>> is huge
>> and
>>> it doesn't seem right to me. Another bad approach would be to use a
>> scanner
>>> starting from the first row and calling Scanner.next() n times  
>>> inside a
>> for
>>> loop to access the n'th row, which also seems very slow. Any ideas  
>>> on how
>> I
>>> could do it more efficiently?
>>>
>>> Thanks in advance,
>>> Jim
>>>
>>


Mime
View raw message