Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: error (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CALHnt0qKZV=4MX2rhJvaPJ18x==oDHedYkwh=WGN7ZDW5ciq4g@mail.gmail.com>
References: 
 <CALHnt0qKZV=4MX2rhJvaPJ18x==oDHedYkwh=WGN7ZDW5ciq4g@mail.gmail.com>
Date: Tue, 24 Feb 2015 09:45:22 +0900
Message-ID: 
 <CAE422GBxddr1QC06G_h9L60wDyenq88v4qZRy2Jv3-OQ2XqhbA@mail.gmail.com>
Subject: Re: tracking remote reads in datanode logs
From: =?UTF-8?B?RHJha2Xrr7zsmIHqt7w=?= <drake.min@nexr.com>
To: user <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=047d7b6760c87153eb050fcad572

--047d7b6760c87153eb050fcad572
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi, Igor

Did you look at the mapreduce application master log? I think the local or
rack local map tasks are logged in the MapReduce AM log.

Good luck.

Drake =EB=AF=BC=EC=98=81=EA=B7=BC Ph.D
kt NexR

On Tue, Feb 24, 2015 at 3:30 AM, Igor Bogomolov <igor.bogomolov@gmail.com>
wrote:

> Hi all,
>
> In a small cluster of 5 nodes that run CDH 5.3.0 (Hadoop 2.5.0) I want to
> know how many remote map tasks (ones that read input data from remote
> nodes) there are in a mapreduce job. For this purpose I took logs of each
> datanode an looked for lines with "op: HDFS_READ" and cliID field that
> contains map task id.
>
> Surprisingly, 4 datanode logs does not contain lines with "op: HDFS_READ"=
.
> Another 1 has many lines with "op: HDFS_READ" but all cliID look like
> DFSClient_NONMAPREDUCE_* and does not contain any map task id.
>
> I concluded there are no remote map tasks but that does not look correct.
> Also even local reads are not logged (because there is no line where cliI=
D
> field contains some map task id). Could anyone please explain what's wron=
g?
> Why logging is not working? (I use default settings).
>
> Chris,
>
> Found HADOOP-3062 <https://issues.apache.org/jira/browse/HADOOP-3062>
> that you have implemented. Thought you might have an explanation.
>
> Best,
> Igor
>
>

--047d7b6760c87153eb050fcad572
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi, Igor<div><br></div><div>Did you look at the mapreduce =
application master log? I think the local or rack local map tasks are logge=
d in the MapReduce AM log.</div><div><br></div><div>Good luck.</div></div><=
div class=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"gmail_signat=
ure"><div dir=3D"ltr"><div><div dir=3D"ltr">Drake =EB=AF=BC=EC=98=81=EA=B7=
=BC Ph.D</div><div>kt NexR</div></div></div></div></div>
<br><div class=3D"gmail_quote">On Tue, Feb 24, 2015 at 3:30 AM, Igor Bogomo=
lov <span dir=3D"ltr">&lt;<a href=3D"mailto:igor.bogomolov@gmail.com" targe=
t=3D"_blank">igor.bogomolov@gmail.com</a>&gt;</span> wrote:<br><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid=
;padding-left:1ex"><div dir=3D"ltr"><div><div><div><div><div><div>Hi all,<b=
r><br></div>In a small cluster of 5 nodes that run <span>CDH</span> 5.3.0 (=
<span>Hadoop</span> 2.5.0) I want to know how many remote map tasks (ones t=
hat read input data from remote nodes) there are in a mapreduce job. For th=
is purpose I took logs of each  <span>datanode</span> an looked for lines w=
ith &quot;op: <span>HDFS</span>_READ&quot; and <span>cliID</span> field tha=
t contains map task id. <br><br></div>Surprisingly, 4 <span>datanode</span>=
 logs does not contain lines with &quot;op: <span>HDFS</span>_READ&quot;. A=
nother 1 has many lines with  &quot;op: <span>HDFS</span>_READ&quot; but al=
l <span>cliID</span> look like <span>DFSClient</span>_<span>NONMAPREDUCE</s=
pan>_* and does not contain any map task id. <br><br>I concluded there are =
no remote map tasks but that does not look correct. Also even local reads a=
re not logged (because there is no line where <span>cliID</span> field cont=
ains some map task id). Could anyone please explain what&#39;s wrong? Why l=
ogging is not working? (I use default settings).<br><br></div>Chris, <br><b=
r></div>Found <a href=3D"https://issues.apache.org/jira/browse/HADOOP-3062"=
 rel=3D"12392029" target=3D"_blank"><span>HADOOP</span>-3062</a> that you h=
ave implemented. Thought you might have an explanation.<br><br></div>Best,<=
br></div>Igor<br><div><div><div><div><br></div></div></div></div></div>
</blockquote></div><br></div>

--047d7b6760c87153eb050fcad572--