hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason hadoop <jason.had...@gmail.com>
Subject Re: ChainMapper/ChainReducer for hbase Tables
Date Tue, 23 Jun 2009 13:27:55 GMT
Can you not just open two output files and output both data sets from one
reduce,either via multi output format or by opening the second find directly
in the configure method of your reduce and closing it in the close?

On Mon, Jun 22, 2009 at 8:27 PM, Irfan Mohammed <irfan.ma@gmail.com> wrote:

> Don't know if this is a M/R question or hbase question but need help in
> solving the following problem.
>
> I have records with some dimensions and some metrics. In this case, lets
> say dimension DD and the metrics [ M1- M2 ].
>
> DD - M1 - M2
> ------------
> ------------
> D1 - N1 - N2
> D2 - N1 - N2
>
> For a given record, I want to store each of the metrics into its own hbase
> table.
>
> T_M1
> ----
>     DD:D1 DD:D2
> R1 -  N1    N1
>
> T_M2
> ----
>     DD:D1 DD:D2
> R1 -  N2    N2
>
> Right now, I have a mapper which converts the plain record into
> <ImmutableBytesWritable, HbaseMapWritable<byte[], byte[]>> and the reducer
> puts into the Hbase table using the <ImmutableBytesWritable, Put>. But the
> process starts all over again for the same record for putting the data into
> the second metrics table.
>
> I want to be able to use ChainMapper/ChainReducer but was not successful.
> ChainReducer allows only one reducer [setReducer] and there can be mappers
> after the reducer. attached is the example code where in the test case, I am
> looping over the whole map/reduce for each table.
>
> Is there a way to be able to do the following?
>
> 1. map
> 2. reduce --> t_m1
> 3. reduce --> t_m2
>
> Thanks,
> Irfan




-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message