hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Shaw <marco.s...@gmail.com>
Subject Re: Monitor network traffic in hadoop
Date Sun, 08 Dec 2013 22:32:33 GMT
Try something like:
http://tutorialgenius.blogspot.ca/2012/03/tcpdump-localhost.html

After you launch tcpdump it mentions "eth0", while your mapred-site.xml is
referring to "localhost" or "lo".  Perhaps...

(Sorry, I don't have a VM running and can't remember if it's "lo" =
"elle-o".)


On Sun, Dec 8, 2013 at 2:57 PM, navaz <navaz.enc@gmail.com> wrote:

> Thanks. I have tried with sudo command and command executed successfully.
> But nothing is captured.
>
> hduser@ubuntu:~/Downloads$ sudo tcpdump port 13562
> [sudo] password for hduser:
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>
> I am running the mapreduce program and this is the output.
>
> hduser@ubuntu:~/Downloads$ hadoop jar chiu-wordcount2.jar WordCount
> /user/hduser/getty/gettysburg.txt /user/hduser/getty/out8
> Warning: $HADOOP_HOME is deprecated.
>
> 13/12/08 10:43:47 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 13/12/08 10:43:47 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/12/08 10:43:47 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 13/12/08 10:43:47 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/12/08 10:43:48 INFO mapred.JobClient: Running job: job_201312081040_0001
> 13/12/08 10:43:49 INFO mapred.JobClient:  map 0% reduce 0%
> 13/12/08 10:43:54 INFO mapred.JobClient:  map 100% reduce 0%
> 13/12/08 10:44:02 INFO mapred.JobClient:  map 100% reduce 33%
> 13/12/08 10:44:03 INFO mapred.JobClient:  map 100% reduce 100%
> 13/12/08 10:44:04 INFO mapred.JobClient: Job complete:
> job_201312081040_0001
> 13/12/08 10:44:04 INFO mapred.JobClient: Counters: 29
> 13/12/08 10:44:04 INFO mapred.JobClient:   Job Counters
> 13/12/08 10:44:04 INFO mapred.JobClient:     Launched reduce tasks=1
> 13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=5478
> 13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all maps
> waiting after reserving slots (ms)=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     Launched map tasks=1
> 13/12/08 10:44:04 INFO mapred.JobClient:     Data-local map tasks=1
> 13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=8804
> 13/12/08 10:44:04 INFO mapred.JobClient:   File Output Format Counters
> 13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Written=1106
> 13/12/08 10:44:04 INFO mapred.JobClient:   FileSystemCounters
> 13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_READ=3059
> 13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_READ=1606
> 13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=108770
> 13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1106
> 13/12/08 10:44:04 INFO mapred.JobClient:   File Input Format Counters
> 13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Read=1486
> 13/12/08 10:44:04 INFO mapred.JobClient:   Map-Reduce Framework
> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output materialized
> bytes=3059
> 13/12/08 10:44:04 INFO mapred.JobClient:     Map input records=6
> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce shuffle bytes=3059
> 13/12/08 10:44:04 INFO mapred.JobClient:     Spilled Records=544
> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output bytes=2509
> 13/12/08 10:44:04 INFO mapred.JobClient:     Total committed heap usage
> (bytes)=131600384
> 13/12/08 10:44:04 INFO mapred.JobClient:     CPU time spent (ms)=1010
> 13/12/08 10:44:04 INFO mapred.JobClient:     Combine input records=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     SPLIT_RAW_BYTES=120
> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input records=272
> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input groups=138
> 13/12/08 10:44:04 INFO mapred.JobClient:     Combine output records=0
> 13/12/08 10:44:04 INFO mapred.JobClient:     Physical memory (bytes)
> snapshot=182116352
> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce output records=138
> 13/12/08 10:44:04 INFO mapred.JobClient:     Virtual memory (bytes)
> snapshot=696483840
> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output records=272
> hduser@ubuntu:~/Downloads$
>
>
>
> mapred-site.xml file is given below.
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>   <name>mapred.job.tracker</name>
>   <value>localhost:54311</value>
>   <description>The host and port that the MapReduce job tracker runs
>   at.  If "local", then jobs are run in-process as a single map
>   and reduce task.
>   </description>
> </property>
> <property>
>   <name>mapreduce.shuffle.port</name>
>   <value>localhost:13562</value>
>   <description>To Monitor traffic on this port
>   </description>
> </property>
> </configuration>
>
>
>
>
> I tried to run again after changing the port number to 54382 still no
> output. Can anybody tell me why I am not able to monitor traffic? Thanks in
> advance.
>
>
> Thanks & Regards
> Abdul Navaz
>
>
> On Fri, Dec 6, 2013 at 12:39 PM, Lars Nilsson <chamaeleon@gmail.com>wrote:
>
>> On Dec 6, 2013 11:08 AM, "navaz" <navaz.enc@gmail.com> wrote:
>> > I have tried tcpdump using "tcpdump port 13562" but not succeeded.
>> Could you please tell me am i following the step correctly.
>> >
>> > hduser@ubuntu:~/Downloads$ tcpdump port 13562
>> > tcpdump: no suitable device found
>>
>> You are probably missing root privileges. Try sudo if you have it on the
>> machine in question, otherwise ask your sysadmin.
>>
>> Lars Nilsson
>>
>
>
>
> --
> *Abdul Navaz*
> *Masters in Network Communications*
> *University of Houston*
> *Houston, TX - 77204-4020*
> *Ph - 281-685-0388 <281-685-0388>*
> *fabdulnavaz@uh.edu* <fabdulnavaz@uh.edu>
>
>

Mime
View raw message