Return-Path: 
 <user-return-22485-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4D1899EF6
	for <apmail-hbase-user-archive@www.apache.org>;
 Mon, 19 Sep 2011 05:35:39 +0000 (UTC)
Received: (qmail 54154 invoked by uid 500); 19 Sep 2011 05:35:37 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 53996 invoked by uid 500); 19 Sep 2011 05:35:37 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 53974 invoked by uid 99); 19 Sep 2011 05:35:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Sep 2011 05:35:36 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [217.196.144.162] (HELO relay02-mailcontrol.conova.com)
 (217.196.144.162)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Sep 2011 05:35:31 +0000
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Writing MR-Job: Something like OracleReducer, JDBCReducer ...
Date: Mon, 19 Sep 2011 07:35:09 +0200
Message-ID: <84B5E4309B3B9F4ABFF7664C3CD7698302D0DD68@kairo.scch.at>
In-Reply-To: 
 <CAKTawb+d4LhfJvQP2YgARQYGA3AFpGGuVRhb_8wQVx8ggd9Yeg@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Writing MR-Job: Something like OracleReducer, JDBCReducer ...
Thread-Index: Acx0irq8m9LxtaJ9Rb+cHR3fsRIcKQCAoWRw
References: 
 <84B5E4309B3B9F4ABFF7664C3CD7698302D0DD64@kairo.scch.at><CAKTawbK9z-Jcxomyp9M7Syif-Vk6w+gdyUipxTd4i-BiQfNbgw@mail.gmail.com><BLU0-SMTP159FC74393CCD0D28561C19F0060@phx.gbl>
 <CAKTawb+d4LhfJvQP2YgARQYGA3AFpGGuVRhb_8wQVx8ggd9Yeg@mail.gmail.com>
From: "Steinmaurer Thomas" <Thomas.Steinmaurer@scch.at>
To: <user@hbase.apache.org>
X-antispameurope-sender: Thomas.Steinmaurer@scch.at
X-antispameurope-recipient: user@hbase.apache.org
X-antispameurope-Virusscan: CLEAN
X-antispameurope: scantime:.3069
X-Virus-Checked: Checked by ClamAV on apache.org

Your assumption is correct. As final output, we want to have aggregated
data in an Oracle database. We are using both, the map and reduce phase.
The row key looks like that:

<datasource-id>-<device-id>-<timestamp>

We basically want to have daily aggregated data, basically measured
values for datasource-id/device-id. We already have a proof-of-concept
implementation, what does exactly that, but as final output, aggregated
data is written into a HBase table again by extending the TableReducer
as our reducer implementation.

See also my thread "MR-Job: Exception in DBOutputFormat".

Thanks again!

Thomas

-----Original Message-----
From: Sonal Goyal [mailto:sonalgoyal4@gmail.com]=20
Sent: Freitag, 16. September 2011 18:07
To: user@hbase.apache.org
Subject: Re: Writing MR-Job: Something like OracleReducer, JDBCReducer
...

Hi Thomas,

I just assumed that you are already using reducers. From what I
understood, please correct me if I am mistaken,

You have data in HBase and you are running a MR job to aggregate the
data.
You have the map as well as reduce phase and as part of the final
output, you want to send the data to Oracle.  is that correct?

Is there any information you would like to share regarding your flow and
data? How big is your data, how often do you need to aggregate, what do
your mappers emit? Are you already using reducers for aggregations?

Best Regards,
Sonal
Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>





On Fri, Sep 16, 2011 at 2:35 PM, Michel Segel
<michael_segel@hotmail.com>wrote:

> I think you need to get a little bit more information.
> Reducers are expensive.
> When Thomas says that he is aggregating data, what exactly does he
mean?
> When dealing w HBase, you really don't want to use a reducer.
>
> You may want to run two map jobs and it could be that just dumping the

> output via jdbc makes the most sense.
>
> We are starting to see a lot of questions where the OP isn't providing

> enough information so that the recommendation could be wrong...
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Sep 16, 2011, at 2:22 AM, Sonal Goyal <sonalgoyal4@gmail.com>
wrote:
>
> > There is a DBOutputFormat class in the=20
> > org.apache,hadoop.mapreduce.lib.db
> > package, you could use that. Or you could write to the hdfs and then

> > use something like HIHO[1] to export to the db. I have been working
> extensively
> > in this area, you can write to me directly if you need any help.
> >
> > 1. https://github.com/sonalgoyal/hiho
> >
> > Best Regards,
> > Sonal
> > Crux: Reporting for HBase <https://github.com/sonalgoyal/crux>
> > Nube Technologies <http://www.nubetech.co>
> >
> > <http://in.linkedin.com/in/sonalgoyal>
> >
> >
> >
> >
> >
> > On Fri, Sep 16, 2011 at 10:55 AM, Steinmaurer Thomas <=20
> > Thomas.Steinmaurer@scch.at> wrote:
> >
> >> Hello,
> >>
> >>
> >>
> >> writing a MR-Job to process HBase data and store aggregated data in

> >> Oracle. How would you do that in a MR-job?
> >>
> >>
> >>
> >> Currently, for test purposes we write the result into a HBase table

> >> again by using a TableReducer. Is there something like a=20
> >> OracleReducer, RelationalReducer, JDBCReducer or whatever? Or=20
> >> should one simply use plan JDBC code in the reduce step?
> >>
> >>
> >>
> >> Thanks!
> >>
> >>
> >>
> >> Thomas
> >>
> >>
> >>
> >>
>