hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Segel, Mike" <mse...@navteq.com>
Subject RE: Dynamic counters in Hadoop M/R jobs...
Date Tue, 20 Jul 2010 01:58:47 GMT
Thanks for the reply.

Got a couple of questions...

I saw that I could get the StatusReporter from the Mapper.Context.
But the API doesn't have a way to get the StatusReport from the Mapper.Context.

If you're going to go to the Mapper.Context you can get the counters.
And you have a getCounter(String, String). Then you can always increment it.

But this kind of begs a question which isn't documented...

If I have a mapper.Context and I say getCounter("Foo","Foobar Count");
What happens if there isn't a counter ("Foo","Foobar Count") ?

What I mean is how do you add a dynamic counter or if I ask for a counter that doesn't already
exist, will it create it?
(This isn't documented... and its not safe to assume anything.)

Sorry, but dynamic counters isn't really well documented.


-----Original Message-----
From: anthony.urso@gmail.com [mailto:anthony.urso@gmail.com] On Behalf Of Anthony Urso
Sent: Monday, July 19, 2010 7:11 PM
To: general@hadoop.apache.org
Subject: Re: Dynamic counters in Hadoop M/R jobs...

The new Mapper class passes a Map.Context object to the map() method.
From this you can get a StatusReporter object  which  can produce a
named Counter object and increment it.


Alternatively, the old APIs are supposed to be temporarily
de-deprecated in the next stable release, so you can use them as
described by that book.

On Mon, Jul 19, 2010 at 11:22 AM, Michael Segel
<michael_segel@hotmail.com> wrote:
> Hi,
> In looking at "The Definitive Guide"  pgs 211-218 looking at the documentation for counters
have a question about how to use the
> dynamic (String, String) methods to create/access counters.
> Looking at the IdentityTableMap class, the following map() method exists():
> public void map(ImmutableBytesWritable key,
>                RowResult value,
>                org.apache.hadoop.mapred.OutputCollector<ImmutableBytesWritable,RowResult>
>                org.apache.hadoop.mapred.Reporter reporter)
>         throws IOExceptionThis method is deprecated in 20.5
> So how can you get the Reporter so you can use the 'reporter.incrCounter(String group,String
counter, long amount)'
> method. (See pg 214 code fragment)
> I must be missing something.
> Thx
> -Mike
> _________________________________________________________________
> The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
> http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4

The information contained in this communication may be CONFIDENTIAL and is intended only for
the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby
notified that any dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this communication in error, please
notify the sender and delete/destroy the original message and any copy of it from your computer
or paper files.
View raw message