Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of arunk786@gmail.com designates
 209.85.213.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <84C0C20E-1523-49D4-BBFD-BFF549FB4B5F@cloudera.com>
References: 
 <CAO7gJeiJwtqUJ4U3FPDu5xN1SAH_q27y-Wd91EeMqf82AUH4iw@mail.gmail.com>
 <84C0C20E-1523-49D4-BBFD-BFF549FB4B5F@cloudera.com>
From: arun k <arunk786@gmail.com>
Date: Sat, 3 Dec 2011 20:00:01 +0530
Message-ID: 
 <CAO7gJei30WNz0JkGddkO6MLVeSHvwiMo6oBvprArFBVhvhTqVw@mail.gmail.com>
Subject: Re: Capturing Map/reduce task run times and bytes read
To: mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf300e4f07fabbe404b330ec2b

--20cf300e4f07fabbe404b330ec2b
Content-Type: text/plain; charset=ISO-8859-1

Harsh,

I wanted to conform about it b'coz in case if it doesn't i want to write
code to capture it.

Does it make sense to classify a map/reduce task as I/O bound or cpu bound
based on its I/O rate ?

Arun

On Sat, Dec 3, 2011 at 2:43 PM, Harsh J <harsh@cloudera.com> wrote:

> Arun,
>
> Inline again.
>
> On 03-Dec-2011, at 12:39 PM, arun k wrote:
>
>
> Q>Does the map/reduce task run time displayed in web GUI is
> decent/accurate enough ?
>
>
> Don't see why not. We only display what's been genuinely collected. What
> you get out of an API on the CLI is absolutely the same thing. Or perhaps I
> do not understand your question completely here - what's led you to ask
> this?
>
> Q>If i want to do find the IO rate of a task, will the task run time
> divided by total number of FIle bytes and HDFS bytes read/written give it
> approximately ?
>
>
> Yes, that should give you a stop-watch measure. Task start -> Task end,
> and the counters the task puts up for itself.
>
> Q>Does the FILE Bytes read for the reduce task include the map output
> record bytes read non-locally over network or the bytes read locally from
> the map output records after they are copied locally ?
>
>
> FILE counters are from whatever is read off a local filesystem (file:///),
> so would mean the latter. If you look again, you will notice another
> counter named "Reduce shuffle bytes" that gives you the former count -
> separately.
>

--20cf300e4f07fabbe404b330ec2b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Harsh,<div><br></div><div>I wanted to conform about it b&#39;coz in case if=
 it doesn&#39;t i want to write code to capture it.</div><div><br></div><di=
v>Does it make sense to classify a map/reduce task as I/O bound or cpu boun=
d based on its I/O rate ?</div>

<div><br></div><div>Arun</div><div><br><div class=3D"gmail_quote">On Sat, D=
ec 3, 2011 at 2:43 PM, Harsh J <span dir=3D"ltr">&lt;<a href=3D"mailto:hars=
h@cloudera.com">harsh@cloudera.com</a>&gt;</span> wrote:<br><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
dding-left:1ex;">

<div style=3D"word-wrap:break-word">Arun,<div><br></div><div>Inline again.<=
/div><div><br><div><div class=3D"im"><div>On 03-Dec-2011, at 12:39 PM, arun=
 k wrote:</div><blockquote type=3D"cite"><div><font color=3D"#000000"><br><=
/font></div>

<div>Q&gt;Does the map/reduce task run time displayed in web GUI is decent/=
accurate enough ?</div></blockquote><div><br></div></div><div>Don&#39;t see=
 why not. We only display what&#39;s been genuinely collected. What you get=
 out of an API on the CLI is absolutely the same thing. Or perhaps I do not=
 understand your question completely here - what&#39;s led you to ask this?=
</div>

<div class=3D"im"><br><blockquote type=3D"cite">

<div>Q&gt;If i want to do find the IO rate of a task, will the task run tim=
e divided by total number of FIle bytes and HDFS bytes read/written give it=
 approximately ?</div></blockquote><div><br></div></div><div>Yes, that shou=
ld give you a stop-watch measure. Task start -&gt; Task end, and the counte=
rs the task puts up for itself.</div>

<div class=3D"im"><br><blockquote type=3D"cite"><div>Q&gt;Does the FILE Byt=
es read for the reduce task include the map output record bytes read non-lo=
cally over network or the bytes read locally from the map output records af=
ter they are copied locally ?</div>

</blockquote><br></div></div></div><div>FILE counters are from whatever is =
read off a local filesystem (<a>file:///</a>), so would mean the latter. If=
 you look again, you will notice another counter named &quot;<span style=3D=
"font-family:sans-serif;font-size:12px;background-color:rgb(255,255,255);di=
splay:inline!important;float:none">Reduce shuffle bytes&quot; that gives yo=
u the former count - separately.</span></div>

</div></blockquote></div><br></div>

--20cf300e4f07fabbe404b330ec2b--