hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From schnitzi <mark.schnitz...@fastsearch.com>
Subject Re: Transactions and Map/Reduce
Date Fri, 26 Jun 2009 00:59:57 GMT

Thanks for the quick reply.  My alternate approach is to just let the mappers
do their writes, and to retrieve values based on the timestamp of the last
*successful* run, which should effectively filter out the results of failed
runs.  It sounds like it may be my only option.

I suppose it might also be possible to generate the output to a separate
table, and then merge it back into the main table only if the m/r finishes
successfully, but that sounds a lot hairier...


Thanks
Mark


Clint Morgan-3 wrote:
> 
> Unfortunately, there is no more documentation on transactions.
> 
> I've never used transactions in M/Rs. You are correct that if you only
> start/commit a transaction in mapper then you will only get atomicity
> across
> the individual map.
> 
> One thing to keep in mind is that, in the current impl, all the writes of
> a
> pending transaction are keep in memory. A big M/R job could quickly blow
> up
> with an OOME.
> 
> -clint
> 
> On Wed, Jun 24, 2009 at 8:45 PM, schnitzi
> <mark.schnitzius@fastsearch.com>wrote:
> 
>>
>> I have some scenarios involving map/reduce that update HBase tables. 
>> (They
>> all involve creating or modifying a number of individual records, so I'm
>> planning to do it all in the mapper.)
>>
>> What I'm hoping to do is to only commit at the end of the map/reduce, if
>> it
>> succeeds.  If I start a TransactionManager transaction in the mapper and
>> commit it there, in my understanding, it will only guarantee atomicity
>> for
>> the set of rows that that mapper updates.
>>
>> Has anyone attempted anything like this before?  Is there a way to do it?
>>
>> Also, is there any general documentation about transactions in HBase
>> besides
>> the javadoc and the test cases?  There doesn't seem to be any on the
>> wiki...
>> I'm muddling through without it but it would be nice to see some.
>>
>>
>> Cheers
>> Mark
>> --
>> View this message in context:
>> http://www.nabble.com/Transactions-and-Map-Reduce-tp24196501p24196501.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/Transactions-and-Map-Reduce-tp24196501p24213337.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message