Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com
 designates 65.55.111.86 as permitted sender)
Message-ID: <BLU0-SMTP3161CFC6E35C0C1E9B141308FE20@phx.gbl>
Content-Type: text/plain; charset="iso-8859-1"
MIME-Version: 1.0 (Apple Message framework v1278)
Subject: Re: Increment Counters in HBase during MapReduce
From: Michael Segel <michael_segel@hotmail.com>
In-Reply-To: 
 <CAE24rAdo+OTGuVXfK5jAHBz-kWZUr4r1k4XVrLMY2WRD5R-E2Q@mail.gmail.com>
Date: Sun, 24 Jun 2012 18:19:11 -0500
Content-Transfer-Encoding: quoted-printable
References: 
 <CAP+k-sf9kAEUnUv1sK2E2temU2_VMYCioood8o2xJ==qn4-P+g@mail.gmail.com>
 <CAGpTDNfS3m3yZqvuqbsKORHBGpHGti_YEJaHGFcTCV=APp1tmQ@mail.gmail.com>
 <CAE24rAdo+OTGuVXfK5jAHBz-kWZUr4r1k4XVrLMY2WRD5R-E2Q@mail.gmail.com>
To: user@hbase.apache.org

There are a couple of issues and I'm sure others will point them out.=20

If you turn off speculative execution on the job, you don't get =
duplicate tasks running in parallel.=20
You could create a table to store your aggregations on a per job basis =
where your row-id could incorporate your job-id.=20
Then at the end of the job. If you didn't have any task failures or =
speculative execution jobs, you could count on your aggregations to be =
correct.=20
If you had a task fail or killed (a simple test if for some reason a job =
ran with speculative execution) you could discard that row's data.=20


On Jun 24, 2012, at 4:15 PM, David Koch wrote:

> Hello J-D
>=20
> I have a similar requirement as that presented by the original poster, =
i.e
> updating a totals count without having to push the entire data set =
through
> the Mapper again.
>=20
> Are you advising against calling incrementColumnValue on a mapper's =
HTable
> instance because the operation is not idempotent or are there other
> reasons? It is even suggested in the docs:
> http://hbase.apache.org/book/mapreduce.example.html (section 7.2.6).
>=20
> Do you know of any "count-exactly-once" implementations on top of =
Hadoop
> Map/Reduce?
>=20
> Thanks,
>=20
> /David
>=20
>=20
> On Tue, Jun 19, 2012 at 6:55 PM, Jean-Daniel Cryans =
<jdcryans@apache.org>wrote:
>=20
>> This question was answered here already:
>>=20
>> =
http://mail-archives.apache.org/mod_mbox/hbase-user/201101.mbox/%3CAANLkTi=
nnW2d7DMCyFu3ptv1Hu_i3XqK_1pDSgD5NT_Lk@mail.gmail.com%3E
>>=20
>> Counters are not idempotent, this can be hard to manage.
>>=20
>> J-D
>>=20
>> On Mon, Jun 18, 2012 at 5:49 PM, Sid Kumar <sqlsid101@gmail.com> =
wrote:
>>> Hi everyone,
>>>=20
>>>   I have a use case in HBase that I was wondering if someone may =
have
>>> stumbled upon. I am maintaining an ad impressions table with columns =
that
>>> are counters for certain metrics. I started using the
>> incrementColumnValue
>>> method part of the HTable API to update these metrics and that works
>> great.
>>>   I was wondering if this function could be used from a MapReduce =
job.
>>> The TableOutputFormat supports only Delete and Put operations. Using =
the
>>> Incremental counters saves me from doing any aggregations in my Map
>> Reduce
>>> code. Ideally i would like to just call this function in my mapper =
and
>>> wouldn't even need a Reducer.
>>>   Has anyone run into this use case? I would also love to know if =
there
>>> are any better alternatives of solving this too. Any info would be =
great.
>>>=20
>>> Thanks
>>> Sid
>>=20