hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From navaz <navaz....@gmail.com>
Subject Re: Monitor network traffic in hadoop
Date Fri, 06 Dec 2013 16:08:00 GMT
Thanks a lot.

As you suggested I have added "mapreduce.shuffle.port" in mpared-site.xml


<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
<property>
  <name>mapreduce.shuffle.port</name>
  <value>localhost:13562</value>
  <description>To Monitor traffic on this port
  </description>
</property>
</configuration>



I have started job again. Installed wireshark and started using "wireshark
&" but it doesnt provide any capture options like interfaces or ports.


I have tried tcpdump using "tcpdump port 13562" but not succeeded. Could
you please tell me am i following the step correctly.

hduser@ubuntu:~/Downloads$ tcpdump port 13562
tcpdump: no suitable device found


Thanks
Abdul Navaz


On Thu, Dec 5, 2013 at 9:31 PM, ch huang <justlooks@gmail.com> wrote:

> hi, Abdul Navaz:
> assign shuffle port in each NM using option "mapreduce.shuffle.port" in
> mapred-site.xml,then
> monitor this port use tcpdump or wireshark ,hope this info can help you
>
> On Fri, Dec 6, 2013 at 11:22 AM, navaz <navaz.enc@gmail.com> wrote:
>
>> Hello
>>
>> I am following the tutorial hadoop on single node cluster and I am able
>> test word count program  map reduce. its working fine.
>>
>> I would like to know
>> How to monitor when shuffle phase network traffic occurs via wireshark or
>> someother means.
>>
>> Pls guide me.
>>
>> Thanks
>> Abdul Navaz
>> Graduate student
>> On Dec 5, 2013 6:56 PM, "navaz" <navaz.enc@gmail.com> wrote:
>>
>>> Hello
>>>
>>> I am following the tutorial hadoop on single node cluster and I am able
>>> test word count map reduce. its working fine.
>>>
>>> I would like to know
>>> How to monitor when shuffle phase network traffic occurs via wireshark
>>> or someother means.
>>>
>>> Pls guide me.
>>>
>>> Thanks
>>> Abdul Navaz
>>> Graduate student
>>> University of Houston ,TX
>>>
>>
>


-- 
*Abdul Navaz*
*Masters in Network Communications*
*University of Houston*
*Houston, TX - 77204-4020*
*Ph - 281-685-0388*
*fabdulnavaz@uh.edu* <fabdulnavaz@uh.edu>

Mime
View raw message