hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason hadoop <jason.had...@gmail.com>
Subject Re: ChainMapper/ChainReducer for hbase Tables
Date Tue, 23 Jun 2009 13:29:14 GMT
Replace file with table in my prior message, and delete the multi output
format reference. I missed that this was hbase-dev

On Tue, Jun 23, 2009 at 6:27 AM, jason hadoop <jason.hadoop@gmail.com>wrote:

> Can you not just open two output files and output both data sets from one
> reduce,either via multi output format or by opening the second find directly
> in the configure method of your reduce and closing it in the close?
>
>
> On Mon, Jun 22, 2009 at 8:27 PM, Irfan Mohammed <irfan.ma@gmail.com>wrote:
>
>> Don't know if this is a M/R question or hbase question but need help in
>> solving the following problem.
>>
>> I have records with some dimensions and some metrics. In this case, lets
>> say dimension DD and the metrics [ M1- M2 ].
>>
>> DD - M1 - M2
>> ------------
>> ------------
>> D1 - N1 - N2
>> D2 - N1 - N2
>>
>> For a given record, I want to store each of the metrics into its own hbase
>> table.
>>
>> T_M1
>> ----
>>     DD:D1 DD:D2
>> R1 -  N1    N1
>>
>> T_M2
>> ----
>>     DD:D1 DD:D2
>> R1 -  N2    N2
>>
>> Right now, I have a mapper which converts the plain record into
>> <ImmutableBytesWritable, HbaseMapWritable<byte[], byte[]>> and the reducer
>> puts into the Hbase table using the <ImmutableBytesWritable, Put>. But the
>> process starts all over again for the same record for putting the data into
>> the second metrics table.
>>
>> I want to be able to use ChainMapper/ChainReducer but was not successful.
>> ChainReducer allows only one reducer [setReducer] and there can be mappers
>> after the reducer. attached is the example code where in the test case, I am
>> looping over the whole map/reduce for each table.
>>
>> Is there a way to be able to do the following?
>>
>> 1. map
>> 2. reduce --> t_m1
>> 3. reduce --> t_m2
>>
>> Thanks,
>> Irfan
>
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message