hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From navaz <navaz....@gmail.com>
Subject Re: Monitor network traffic in hadoop
Date Mon, 09 Dec 2013 01:52:15 GMT
Yes. I have tried with *tcpdump -i lo*      and   * tshark -i lo* and i am
able to monitor this

17:38:55.553775 IP localhost.54311 > localhost.39619: Flags [P.], seq
1499:1606, ack 4095, win 1020, options [nop,nop,TS val 446136 ecr 446136],
length 107
17:38:55.553787 IP localhost.39619 > localhost.54311: Flags [.], ack 1606,
win 685, options [nop,nop,TS val 446136 ecr 446136], length 0
17:38:55.855535 IP localhost.39619 > localhost.54311: Flags [P.], seq
4095:4368, ack 1606, win 685, options [nop,nop,TS val 446211 ecr 446136],
length 273
17:38:55.855839 IP localhost.54311 > localhost.39619: Flags [P.], seq
1606:1713, ack 4368, win 1020, options [nop,nop,TS val 446211 ecr 446211],
length 107
17:38:55.856061 IP localhost.39619 > localhost.54311: Flags [.], ack 1713,
win 685, options [nop,nop,TS val 446212 ecr 446211], length 0
17:38:56.102755 IP localhost.37150 > localhost.domain: 1+ TXT?
_kerberos.localdomain. (39)
17:38:56.104445 IP localhost.domain > localhost.37150: 1 NXDomain 0/1/0
(114)
17:38:56.157408 IP localhost.39619 > localhost.54311: Flags [P.], seq
4368:4641, ack 1713, win 685, options [nop,nop,TS val 446287 ecr 446211],
length 2



I am not able to see the port number ( 13562)  in the logs which i have
mentioned in mapred.xml file.
Also   tcpdump -i lo port 13562 doesn't capture anything.




Thanks
Navaz



On Sun, Dec 8, 2013 at 4:32 PM, Marco Shaw <marco.shaw@gmail.com> wrote:

> Try something like:
> http://tutorialgenius.blogspot.ca/2012/03/tcpdump-localhost.html
>
> After you launch tcpdump it mentions "eth0", while your mapred-site.xml is
> referring to "localhost" or "lo".  Perhaps...
>
> (Sorry, I don't have a VM running and can't remember if it's "lo" =
> "elle-o".)
>
>
> On Sun, Dec 8, 2013 at 2:57 PM, navaz <navaz.enc@gmail.com> wrote:
>
>> Thanks. I have tried with sudo command and command executed successfully.
>> But nothing is captured.
>>
>> hduser@ubuntu:~/Downloads$ sudo tcpdump port 13562
>> [sudo] password for hduser:
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>>
>> I am running the mapreduce program and this is the output.
>>
>> hduser@ubuntu:~/Downloads$ hadoop jar chiu-wordcount2.jar WordCount
>> /user/hduser/getty/gettysburg.txt /user/hduser/getty/out8
>> Warning: $HADOOP_HOME is deprecated.
>>
>> 13/12/08 10:43:47 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>> 13/12/08 10:43:47 INFO input.FileInputFormat: Total input paths to
>> process : 1
>> 13/12/08 10:43:47 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 13/12/08 10:43:47 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 13/12/08 10:43:48 INFO mapred.JobClient: Running job:
>> job_201312081040_0001
>> 13/12/08 10:43:49 INFO mapred.JobClient:  map 0% reduce 0%
>> 13/12/08 10:43:54 INFO mapred.JobClient:  map 100% reduce 0%
>> 13/12/08 10:44:02 INFO mapred.JobClient:  map 100% reduce 33%
>> 13/12/08 10:44:03 INFO mapred.JobClient:  map 100% reduce 100%
>> 13/12/08 10:44:04 INFO mapred.JobClient: Job complete:
>> job_201312081040_0001
>> 13/12/08 10:44:04 INFO mapred.JobClient: Counters: 29
>> 13/12/08 10:44:04 INFO mapred.JobClient:   Job Counters
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Launched reduce tasks=1
>> 13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=5478
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Launched map tasks=1
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Data-local map tasks=1
>> 13/12/08 10:44:04 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=8804
>> 13/12/08 10:44:04 INFO mapred.JobClient:   File Output Format Counters
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Written=1106
>> 13/12/08 10:44:04 INFO mapred.JobClient:   FileSystemCounters
>> 13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_READ=3059
>> 13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_READ=1606
>> 13/12/08 10:44:04 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=108770
>> 13/12/08 10:44:04 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1106
>> 13/12/08 10:44:04 INFO mapred.JobClient:   File Input Format Counters
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Bytes Read=1486
>> 13/12/08 10:44:04 INFO mapred.JobClient:   Map-Reduce Framework
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output materialized
>> bytes=3059
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Map input records=6
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce shuffle bytes=3059
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Spilled Records=544
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output bytes=2509
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Total committed heap usage
>> (bytes)=131600384
>> 13/12/08 10:44:04 INFO mapred.JobClient:     CPU time spent (ms)=1010
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Combine input records=0
>> 13/12/08 10:44:04 INFO mapred.JobClient:     SPLIT_RAW_BYTES=120
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input records=272
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce input groups=138
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Combine output records=0
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Physical memory (bytes)
>> snapshot=182116352
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Reduce output records=138
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Virtual memory (bytes)
>> snapshot=696483840
>> 13/12/08 10:44:04 INFO mapred.JobClient:     Map output records=272
>> hduser@ubuntu:~/Downloads$
>>
>>
>>
>> mapred-site.xml file is given below.
>>
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>>
>> <configuration>
>> <property>
>>   <name>mapred.job.tracker</name>
>>   <value>localhost:54311</value>
>>   <description>The host and port that the MapReduce job tracker runs
>>   at.  If "local", then jobs are run in-process as a single map
>>   and reduce task.
>>   </description>
>> </property>
>> <property>
>>   <name>mapreduce.shuffle.port</name>
>>   <value>localhost:13562</value>
>>   <description>To Monitor traffic on this port
>>   </description>
>> </property>
>> </configuration>
>>
>>
>>
>>
>> I tried to run again after changing the port number to 54382 still no
>> output. Can anybody tell me why I am not able to monitor traffic? Thanks in
>> advance.
>>
>>
>> Thanks & Regards
>> Abdul Navaz
>>
>>
>> On Fri, Dec 6, 2013 at 12:39 PM, Lars Nilsson <chamaeleon@gmail.com>wrote:
>>
>>> On Dec 6, 2013 11:08 AM, "navaz" <navaz.enc@gmail.com> wrote:
>>> > I have tried tcpdump using "tcpdump port 13562" but not succeeded.
>>> Could you please tell me am i following the step correctly.
>>> >
>>> > hduser@ubuntu:~/Downloads$ tcpdump port 13562
>>> > tcpdump: no suitable device found
>>>
>>> You are probably missing root privileges. Try sudo if you have it on the
>>> machine in question, otherwise ask your sysadmin.
>>>
>>> Lars Nilsson
>>>
>>
>>
>>
>> --
>> *Abdul Navaz*
>> *Masters in Network Communications*
>> *University of Houston*
>> *Houston, TX - 77204-4020*
>> *Ph - 281-685-0388 <281-685-0388>*
>> *fabdulnavaz@uh.edu* <fabdulnavaz@uh.edu>
>>
>>
>


-- 
*Abdul Navaz*
*Masters in Network Communications*
*University of Houston*
*Houston, TX - 77204-4020*
*Ph - 281-685-0388*
*fabdulnavaz@uh.edu* <fabdulnavaz@uh.edu>

Mime
View raw message