Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
MIME-Version: 1.0
In-Reply-To: <5448E769.8090907@gmail.com>
References: <5448E769.8090907@gmail.com>
Date: Thu, 23 Oct 2014 14:56:24 +0200
Message-ID: 
 <CALSJUsQxSTd0y9QVkciEq+Gf80HB_CugAQTfB=JEH4b5cYbLig@mail.gmail.com>
Subject: Re: getting counters from specific hadoop jobs
From: Dieter De Witte <drdwitte@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11c341e07dcfe505061697a7

--001a11c341e07dcfe505061697a7
Content-Type: text/plain; charset=UTF-8

Maybe you could use job -list or job -history to get a list of the jobids
and extract it from there?

2014-10-23 13:32 GMT+02:00 Bart Vandewoestyne <Bart.Vandewoestyne@telenet.be
>:

> Hello list,
>
> I order to learn about Hadoop performance tuning, I am currently
> investigating the effect of certain Hadoop configuration parameters on
> certain Hadoop counters.  I would like to do something like the following
> (from the command line):
>
> for some_config_parameter in set_of_config_values
>
>   Step 1) run hadoop job with 'hadoop jar ....'
>
>   Step 2) once job finished, get the value of one or more Hadoop counters
> of this job
>
> I know that I can achieve step 2 with the -counter option of the mapred
> job command:
>
> bart@sandy-quad-1:~$ mapred job -counter
> Usage: CLI [-counter <job-id> <group-name> <counter-name>]
>
> However, I need to specify a job-id here, and that is where I'm having
> trouble... I don't know an easy way to get the job-id from the hadoop job
> that I started in Step 1.  I also don't know of a way to specify a job-id
> myself in Step 1 so that I can use it later in Step 2.
>
> I cannot imagine I'm the only one trying to run jobs and requesting some
> of the counters afterwards.  How is this typically solved?
>
> Note that I'm looking for a command-line solution, something that is
> scriptable bash or so.
>
> Thanks,
> Bart
>

--001a11c341e07dcfe505061697a7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Maybe you could use job -list or job -history to get a lis=
t of the jobids and extract it from there?<br></div><div class=3D"gmail_ext=
ra"><br><div class=3D"gmail_quote">2014-10-23 13:32 GMT+02:00 Bart Vandewoe=
styne <span dir=3D"ltr">&lt;<a href=3D"mailto:Bart.Vandewoestyne@telenet.be=
" target=3D"_blank">Bart.Vandewoestyne@telenet.be</a>&gt;</span>:<br><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
 solid;padding-left:1ex">Hello list,<br>
<br>
I order to learn about Hadoop performance tuning, I am currently investigat=
ing the effect of certain Hadoop configuration parameters on certain Hadoop=
 counters.=C2=A0 I would like to do something like the following (from the =
command line):<br>
<br>
for some_config_parameter in set_of_config_values<br>
<br>
=C2=A0 Step 1) run hadoop job with &#39;hadoop jar ....&#39;<br>
<br>
=C2=A0 Step 2) once job finished, get the value of one or more Hadoop count=
ers of this job<br>
<br>
I know that I can achieve step 2 with the -counter option of the mapred job=
 command:<br>
<br>
bart@sandy-quad-1:~$ mapred job -counter<br>
Usage: CLI [-counter &lt;job-id&gt; &lt;group-name&gt; &lt;counter-name&gt;=
]<br>
<br>
However, I need to specify a job-id here, and that is where I&#39;m having =
trouble... I don&#39;t know an easy way to get the job-id from the hadoop j=
ob that I started in Step 1.=C2=A0 I also don&#39;t know of a way to specif=
y a job-id myself in Step 1 so that I can use it later in Step 2.<br>
<br>
I cannot imagine I&#39;m the only one trying to run jobs and requesting som=
e of the counters afterwards.=C2=A0 How is this typically solved?<br>
<br>
Note that I&#39;m looking for a command-line solution, something that is sc=
riptable bash or so.<br>
<br>
Thanks,<br>
Bart<br>
</blockquote></div><br></div>

--001a11c341e07dcfe505061697a7--