Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of buttler1@llnl.gov designates
 128.115.41.82 as permitted sender)
From: "Buttler, David" <buttler1@llnl.gov>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Date: Fri, 25 Mar 2011 10:00:26 -0700
Subject: RE: How could I re-calculate every entries in hbase efficiently
 through mapreduce?
Thread-Topic: How could I re-calculate every entries in hbase efficiently
 through mapreduce?
Thread-Index: AcvrBRrUla6QQsqQQ96OWS/35Pwm0QACM15w
Message-ID: 
 <2D6136772A13B84E95DF6DA79E85A9F0013AD7FB8E53@NSPEXMBX-A.the-lab.llnl.gov>
References: 
 <AANLkTimcrikrj=hD6BBkO5352pEqKK=BgHHHk===Nci1@mail.gmail.com>,<2D6136772A13B84E95DF6DA79E85A9F0013AD7FB8DF2@NSPEXMBX-A.the-lab.llnl.gov>,<COL117-W4881C66969110B3BEFF2538FB90@phx.gbl>,<2D6136772A13B84E95DF6DA79E85A9F0013AD7FB8E03@NSPEXMBX-A.the-lab.llnl.gov>
 <COL117-W80FB201F6EA4BF80163798FB90@phx.gbl>
In-Reply-To: <COL117-W80FB201F6EA4BF80163798FB90@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I would certainly find it useful if you wrote such a blog post.
Dave

-----Original Message-----
From: Michael Segel [mailto:michael_segel@hotmail.com]=20
Sent: Friday, March 25, 2011 8:55 AM
To: user@hbase.apache.org
Subject: RE: How could I re-calculate every entries in hbase efficiently th=
rough mapreduce?


"During inserts into the table, there was one field that was populated=20
from hand-crafted HTML that should only have a small range of values=20
(e.g. a primary color). We wanted to keep a log of all of the unique=20
values that were found here, and so the values were the map job output=20
and then sorted and counted in the reduce phase."

Ahhh, have you heard about dynamic counters?
You don't need a reducer and all you have to do is dump the counters in you=
r main job after your mappers run.

Maybe I should write a blog entry where you can do your word counter app us=
ing just dynamic counters and no reducers?

HTH

-Mike


----------------------------------------
> From: buttler1@llnl.gov
> To: user@hbase.apache.org
> Date: Fri, 25 Mar 2011 08:44:12 -0700
> Subject: RE: How could I re-calculate every entries in hbase efficiently =
through mapreduce?
>
> We ran across a use-case this week. During inserts into the table, there =
was one field that was populated from hand-crafted HTML that should only ha=
ve a small range of values (e.g. a primary color). We wanted to keep a log =
of all of the unique values that were found here, and so the values were th=
e map job output and then sorted and counted in the reduce phase. A handy w=
ay for us to debug the HTML into a persistent file (we could have just used=
 counters, but those disappear after a while unless you manually copy them)=
.
>
> -----Original Message-----
> From: Michael Segel [mailto:michael_segel@hotmail.com]
> Sent: Friday, March 25, 2011 8:26 AM
> To: user@hbase.apache.org
> Subject: RE: How could I re-calculate every entries in hbase efficiently =
through mapreduce?
>
>
>
> Yeah...
> Uhm I don't know of many use cases where you would want or need a reducer=
 step when dealing with HBase.
> I'm sure one may exist, but from past practical experience... you shouldn=
't need one.
>
> ----------------------------------------
> > From: buttler1@llnl.gov
> > To: user@hbase.apache.org
> > Date: Fri, 25 Mar 2011 08:20:45 -0700
> > Subject: RE: How could I re-calculate every entries in hbase efficientl=
y through mapreduce?
> >
> > There is no reason to use a reducer in this scenario. I frequently do m=
ap-only update jobs. Skipping the reduce step saves a lot of unnecessary wo=
rk.
> >
> > Dave
> >
> > -----Original Message-----
> > From: Stanley Xu [mailto:wenhao.xu@gmail.com]
> > Sent: Thursday, March 24, 2011 7:37 PM
> > To: user@hbase.apache.org
> > Subject: How could I re-calculate every entries in hbase efficiently th=
rough mapreduce?
> >
> > Dear Buddies,
> >
> > I need to re-calculate the entries in a hbase everyday, like let x =3D =
0.9x
> > everyday, to make the time has impact on the entry values.
> >
> > So I write a TableMapper to get the Entry, and recalculate the result, =
and
> > use Context.write(key, put) to put the update operation in context, and=
 then
> > use a IdentityTableReducer to write that directly back the hbase. In or=
der
> > to make the job done in a short time, I use the HRegionPartitioner to
> > increase the reducer number to 50.
> >
> > But I have two doubts here:
> > 1. It looks the partitioner will do a lots of shuffling, I am wondering=
 why
> > it couldn't just do the put on the local region since the read and writ=
e on
> > the same entry should be on the same region, isn't it?
> >
> > 2. If the job failed for any reason(like timeout), the HBase might be i=
n a
> > partial-updated status, is it?
> >
> > Is there any suggestion that I could avoid these two problems?
> >
> >
> > Thanks.
> >
> > Best wishes,
> > Stanley Xu
>
 		 	   		 =20