hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From navaz <navaz....@gmail.com>
Subject Re: Monitor network traffic in hadoop
Date Sun, 08 Dec 2013 18:57:58 GMT
Thanks. I have tried with sudo command and command executed successfully.
But nothing is captured.

hduser@ubuntu:~/Downloads$ sudo tcpdump port 13562
[sudo] password for hduser:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

I am running the mapreduce program and this is the output.

hduser@ubuntu:~/Downloads$ hadoop jar chiu-wordcount2.jar WordCount
/user/hduser/getty/gettysburg.txt /user/hduser/getty/out8
Warning: $HADOOP_HOME is deprecated.

13/12/08 10:43:47 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
13/12/08 10:43:47 INFO input.FileInputFormat: Total input paths to process
: 1
13/12/08 10:43:47 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
13/12/08 10:43:47 WARN snappy.LoadSnappy: Snappy native library not loaded
13/12/08 10:43:48 INFO mapred.JobClient: Running job: job_201312081040_0001
13/12/08 10:43:49 INFO mapred.JobClient:  map 0% reduce 0%
13/12/08 10:43:54 INFO mapred.JobClient:  map 100% reduce 0%
13/12/08 10:44:02 INFO mapred.JobClient:  map 100% reduce 33%
13/12/08 10:44:03 INFO mapred.JobClient:  map 100% reduce 100%
13/12/08 10:44:04 INFO mapred.JobClient: Job complete: job_201312081040_0001
13/12/08 10:44:04 INFO mapred.JobClient: Counters: 29
13/12/08 10:44:04 INFO mapred.JobClient:   Job Counters
13/12/08 10:44:04 INFO mapred.JobClient:     Launched reduce tasks=1
13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=5478
13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all
reduces waiting after reserving slots (ms)=0
13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all maps
waiting after reserving slots (ms)=0
13/12/08 10:44:04 INFO mapred.JobClient:     Launched map tasks=1
13/12/08 10:44:04 INFO mapred.JobClient:     Data-local map tasks=1
13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=8804
13/12/08 10:44:04 INFO mapred.JobClient:   File Output Format Counters
13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Written=1106
13/12/08 10:44:04 INFO mapred.JobClient:   FileSystemCounters
13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_READ=3059
13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_READ=1606
13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=108770
13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1106
13/12/08 10:44:04 INFO mapred.JobClient:   File Input Format Counters
13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Read=1486
13/12/08 10:44:04 INFO mapred.JobClient:   Map-Reduce Framework
13/12/08 10:44:04 INFO mapred.JobClient:     Map output materialized
bytes=3059
13/12/08 10:44:04 INFO mapred.JobClient:     Map input records=6
13/12/08 10:44:04 INFO mapred.JobClient:     Reduce shuffle bytes=3059
13/12/08 10:44:04 INFO mapred.JobClient:     Spilled Records=544
13/12/08 10:44:04 INFO mapred.JobClient:     Map output bytes=2509
13/12/08 10:44:04 INFO mapred.JobClient:     Total committed heap usage
(bytes)=131600384
13/12/08 10:44:04 INFO mapred.JobClient:     CPU time spent (ms)=1010
13/12/08 10:44:04 INFO mapred.JobClient:     Combine input records=0
13/12/08 10:44:04 INFO mapred.JobClient:     SPLIT_RAW_BYTES=120
13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input records=272
13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input groups=138
13/12/08 10:44:04 INFO mapred.JobClient:     Combine output records=0
13/12/08 10:44:04 INFO mapred.JobClient:     Physical memory (bytes)
snapshot=182116352
13/12/08 10:44:04 INFO mapred.JobClient:     Reduce output records=138
13/12/08 10:44:04 INFO mapred.JobClient:     Virtual memory (bytes)
snapshot=696483840
13/12/08 10:44:04 INFO mapred.JobClient:     Map output records=272
hduser@ubuntu:~/Downloads$



mapred-site.xml file is given below.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
<property>
  <name>mapreduce.shuffle.port</name>
  <value>localhost:13562</value>
  <description>To Monitor traffic on this port
  </description>
</property>
</configuration>




I tried to run again after changing the port number to 54382 still no
output. Can anybody tell me why I am not able to monitor traffic? Thanks in
advance.


Thanks & Regards
Abdul Navaz


On Fri, Dec 6, 2013 at 12:39 PM, Lars Nilsson <chamaeleon@gmail.com> wrote:

> On Dec 6, 2013 11:08 AM, "navaz" <navaz.enc@gmail.com> wrote:
> > I have tried tcpdump using "tcpdump port 13562" but not succeeded. Could
> you please tell me am i following the step correctly.
> >
> > hduser@ubuntu:~/Downloads$ tcpdump port 13562
> > tcpdump: no suitable device found
>
> You are probably missing root privileges. Try sudo if you have it on the
> machine in question, otherwise ask your sysadmin.
>
> Lars Nilsson
>



-- 
*Abdul Navaz*
*Masters in Network Communications*
*University of Houston*
*Houston, TX - 77204-4020*
*Ph - 281-685-0388*
*fabdulnavaz@uh.edu* <fabdulnavaz@uh.edu>

Mime
View raw message