Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
Mime-Version: 1.0 (Apple Message framework v1278)
Content-Type: multipart/signed;
 boundary="Apple-Mail=_46B971DF-53FD-48C3-8044-FF8A96D8DA4D";
 protocol="application/pgp-signature"; micalg=pgp-sha1
Subject: Re: Counters across all jobs
Date: Mon, 10 Sep 2012 10:46:13 -0700
In-Reply-To: 
 <CAJUrEoYaD9k=8AkcYQNH31h0rOBeCQgkvx0omdipsM_zY=t3yA@mail.gmail.com>
To: mapreduce-user@hadoop.apache.org
References: 
 <CAJUrEoYaD9k=8AkcYQNH31h0rOBeCQgkvx0omdipsM_zY=t3yA@mail.gmail.com>
Message-Id: <B45BB585-6DE4-4FB0-99DE-8D645F746428@apache.org>


--Apple-Mail=_46B971DF-53FD-48C3-8044-FF8A96D8DA4D
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_E223EF42-D955-4CD3-940F-79881AB49275"


--Apple-Mail=_E223EF42-D955-4CD3-940F-79881AB49275
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1


Counters are per-job in Hadoop MapReduce. You need an external =
aggregator for such cross-job counters - for e.g. a node in Zookeeper.

Also, is it just for display or your job-logic depends on this? If it is =
the earlier, and if you don't have a problem with waiting till jobs =
finish, you can do a post-process on the counters of all jobs and =
calculate the aggregates.

Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Aug 28, 2012, at 1:20 AM, Kasi Subrahmanyam wrote:

> Hi,
>=20
> I have around 4 jobs running in a controller.
> How can i have a single unique counter present in all the jobs and =
incremented where ever used in a job?
>=20
> For example:Consider a counter ACount.
> If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6.
> Can i have the  counter displayed output in the jobtracker as
> job1:2
> job2:2
> job3:7
> job4:13
>=20
> Thanks,
> Subbu
>=20


--Apple-Mail=_E223EF42-D955-4CD3-940F-79881AB49275
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=iso-8859-1

<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div>Counters are per-job in Hadoop MapReduce. You need an external aggregator for such cross-job counters - for e.g. a node in Zookeeper.<div><br></div><div>Also, is it just for display or your job-logic depends on this? If it is the earlier, and if you don't have a problem with waiting till jobs finish, you can do a post-process on the counters of all jobs and calculate the aggregates.</div><div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><div class="">Thanks,</div><div class="">+Vinod Kumar Vavilapalli</div><div class="">Hortonworks Inc.<br class=""><a href="http://hortonworks.com/" class="">http://hortonworks.com/</a></div></div></span>
</div>

<br><div><div>On Aug 28, 2012, at 1:20 AM, Kasi Subrahmanyam wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Hi,<div><br></div><div>I have around 4 jobs running in a controller.</div><div>How can i have a single unique counter present in all the jobs and incremented&nbsp;where ever used in a job?</div><div><br></div><div>For example:Consider a counter ACount.</div>
<div>If job1 is incrementing the counter by2 and job3 by 5 and job 4 by 6.</div><div>Can i have the &nbsp;counter displayed output in the jobtracker as</div><div>job1:2</div><div>job2:2</div><div>job3:7</div><div>job4:13</div>
<div><br></div><div>Thanks,</div><div>Subbu</div><div><br></div>
</blockquote></div><br></div></body></html>
--Apple-Mail=_E223EF42-D955-4CD3-940F-79881AB49275--

--Apple-Mail=_46B971DF-53FD-48C3-8044-FF8A96D8DA4D
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)

iQEcBAEBAgAGBQJQTidlAAoJECVa31bDbF8PtUkIALQ3JnZRq7jGvXXJqIwn9Xj1
xtEfbuvTnXKutLfQwFEm6PtCMxq1/+BqDhm/EPqXGfcChg44MP3Ro8bnaJLksml7
ImdLTobtdCRsn7VFuQzjUT1VNQ8YPlXSJmG3a3UZ1n/Upfgq7Ss5bSkUfPAXO1ut
87NvpsWd7lV6zSdc6RIhs+apiwevmqU7aaJo0ZFsftxSAiiiRg0lhmR8nCP8SK15
XWts/a7852Y6dRO99gFBgqcL11iocjMn2WR8/HKO98S9jP5PGCOfVHssUdA8lS8n
k7gFnDZdhpcHmN//HdRFcAzWjQX4YKZXomB4Rp0jbHhi1rr4nzcTzXUOBayVjaQ=
=hVQ+
-----END PGP SIGNATURE-----

--Apple-Mail=_46B971DF-53FD-48C3-8044-FF8A96D8DA4D--