Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: 209.85.223.176 is neither permitted
 nor denied by domain of jeremy@lewi.us)
MIME-Version: 1.0
In-Reply-To: 
 <CAOcnVr0Cv04SundCUVJk+DyDijUbj_XyVpieYk7mjDB5EpTWMw@mail.gmail.com>
References: 
 <CACijwibU-UeZTZfYtHPRPi_JF3m3_BUyUPPdk_1XwZKJ2i5GeA@mail.gmail.com>
	<CAOcnVr0Cv04SundCUVJk+DyDijUbj_XyVpieYk7mjDB5EpTWMw@mail.gmail.com>
Date: Fri, 5 Oct 2012 08:48:58 -0700
Message-ID: 
 <CACijwiZLii6+_2RqV=nexK-v3bP2_ef2L1uS7A87e3duHRRC=A@mail.gmail.com>
Subject: Re: Counters that track the max value
From: Jeremy Lewi <jeremy@lewi.us>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=14dae93408cf607e5304cb51cf0d

--14dae93408cf607e5304cb51cf0d
Content-Type: text/plain; charset=ISO-8859-1

HI Harsh,

Thank you very much that will work.

How come we can't simply create a modification of a regular mapreduce
counter which does this behind the scenes? It seems like we should just be
able to replace "+" with "max" and everything else should work?

J

On Wed, Oct 3, 2012 at 9:52 AM, Harsh J <harsh@cloudera.com> wrote:

> Jeremy,
>
> Here's my shot at it (pardon the quick crappy code):
> https://gist.github.com/3828246
>
> Basically - you can achieve it in two ways:
>
> Requirement:  All tasks must increment the "max" designated counter
> only AFTER the max has been computed (i.e. in cleanup).
>
> 1. All tasks may use same counter name. Later, we pull per-task
> counters and determine the max at the client. (This is my quick and
> dirty implementation)
> 2. All tasks may use their own task ID (Number part) in the counter
> name, but use the same group. Later, we fetch all counters for that
> group and iterate over it to find the max. This is cleaner, and
> doesn't end up using deprecated APIs such as the above.
>
> Does this help?
>
> On Wed, Oct 3, 2012 at 8:47 PM, Jeremy Lewi <jeremy@lewi.us> wrote:
> > HI hadoop-users,
> >
> > I'm curious if there is an implementation somewhere of a counter which
> > tracks the maximum of some value across all mappers or reducers?
> >
> > Thanks
> > J
>
>
>
> --
> Harsh J
>

--14dae93408cf607e5304cb51cf0d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

HI Harsh,<div><br></div><div>Thank you very much that will work.=A0</div><d=
iv><br></div><div>How come we can&#39;t simply create a modification of a r=
egular mapreduce counter which does this behind the scenes? It seems like w=
e should just be able to replace &quot;+&quot; with &quot;max&quot; and eve=
rything else should work?</div>
<div><br></div><div>J<br><br><div class=3D"gmail_quote">On Wed, Oct 3, 2012=
 at 9:52 AM, Harsh J <span dir=3D"ltr">&lt;<a href=3D"mailto:harsh@cloudera=
.com" target=3D"_blank">harsh@cloudera.com</a>&gt;</span> wrote:<br><blockq=
uote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex">
Jeremy,<br>
<br>
Here&#39;s my shot at it (pardon the quick crappy code):<br>
<a href=3D"https://gist.github.com/3828246" target=3D"_blank">https://gist.=
github.com/3828246</a><br>
<br>
Basically - you can achieve it in two ways:<br>
<br>
Requirement: =A0All tasks must increment the &quot;max&quot; designated cou=
nter<br>
only AFTER the max has been computed (i.e. in cleanup).<br>
<br>
1. All tasks may use same counter name. Later, we pull per-task<br>
counters and determine the max at the client. (This is my quick and<br>
dirty implementation)<br>
2. All tasks may use their own task ID (Number part) in the counter<br>
name, but use the same group. Later, we fetch all counters for that<br>
group and iterate over it to find the max. This is cleaner, and<br>
doesn&#39;t end up using deprecated APIs such as the above.<br>
<br>
Does this help?<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
On Wed, Oct 3, 2012 at 8:47 PM, Jeremy Lewi &lt;<a href=3D"mailto:jeremy@le=
wi.us">jeremy@lewi.us</a>&gt; wrote:<br>
&gt; HI hadoop-users,<br>
&gt;<br>
&gt; I&#39;m curious if there is an implementation somewhere of a counter w=
hich<br>
&gt; tracks the maximum of some value across all mappers or reducers?<br>
&gt;<br>
&gt; Thanks<br>
&gt; J<br>
<br>
<br>
<br>
</div></div><span class=3D"HOEnZb"><font color=3D"#888888">--<br>
Harsh J<br>
</font></span></blockquote></div><br></div>

--14dae93408cf607e5304cb51cf0d--