hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From EdwardKing <zhan...@neusoft.com>
Subject Which node is used
Date Mon, 10 Feb 2014 08:44:10 GMT
I have two node,one is master(172.11.12.6) and one is node1(172.11.12.7), I use following command
to run a program under 172.11.12.6 ,like follows:
$ hadoop jar wc2.jar WordCountPredefined abc.txt output

[hadoop@master ~]$ hadoop jar wc2.jar WordCountPredefined abc.txt output
14/02/10 00:23:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
14/02/10 00:23:25 INFO client.RMProxy: Connecting to ResourceManager at master/172.11.12.6:8993
14/02/10 00:23:26 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed.
Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/02/10 00:23:28 INFO input.FileInputFormat: Total input paths to process : 1
14/02/10 00:23:28 INFO mapreduce.JobSubmitter: number of splits:1
14/02/10 00:23:28 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.output.value.class is deprecated.
Instead, use mapreduce.job.output.value.class
14/02/10 00:23:28 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead,
use mapreduce.job.map.class
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead,
use mapreduce.job.name
14/02/10 00:23:28 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead,
use mapreduce.job.reduce.class
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead,
use mapreduce.input.fileinputformat.inputdir
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead,
use mapreduce.output.fileoutputformat.outputdir
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead,
use mapreduce.job.maps
14/02/10 00:23:28 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead,
use mapreduce.job.output.key.class
14/02/10 00:23:29 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead,
use mapreduce.job.working.dir
14/02/10 00:23:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1391993915383_0003
14/02/10 00:23:30 INFO impl.YarnClientImpl: Submitted application application_1391993915383_0003
to ResourceManager at master/172.11.12.6:8993
14/02/10 00:23:31 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1391993915383_0003/
14/02/10 00:23:31 INFO mapreduce.Job: Running job: job_1391993915383_0003
14/02/10 00:23:54 INFO mapreduce.Job: Job job_1391993915383_0003 running in uber mode : false
14/02/10 00:23:54 INFO mapreduce.Job:  map 0% reduce 0%
14/02/10 00:26:09 INFO mapreduce.Job:  map 67% reduce 0%
14/02/10 00:26:12 INFO mapreduce.Job:  map 100% reduce 0%
14/02/10 00:26:43 INFO mapreduce.Job:  map 100% reduce 100%
14/02/10 00:26:45 INFO mapreduce.Job: Job job_1391993915383_0003 completed successfully
14/02/10 00:26:46 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=30
                FILE: Number of bytes written=156945
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=115
                HDFS: Number of bytes written=16
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=139690
                Total time spent by all reduces in occupied slots (ms)=20127
        Map-Reduce Framework
                Map input records=1
                Map output records=2
                Map output bytes=20
                Map output materialized bytes=30
                Input split bytes=103
                Combine input records=0
                Combine output records=0
                Reduce input groups=2
                Reduce shuffle bytes=30
                Reduce input records=2
                Reduce output records=2
                Spilled Records=4
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=396
                CPU time spent (ms)=2130
                Physical memory (bytes) snapshot=199540736
                Virtual memory (bytes) snapshot=709742592
                Total committed heap usage (bytes)=124510208
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=12
        File Output Format Counters 
                Bytes Written=16

I can view http://172.11.12.6:9002/dfsnodelist.jsp?whatNodes=LIVE in firefox

NameNode 'master:9000'
Started: Sun Feb 09 16:57:50 PST 2014
Version: 2.2.0, 1529768
Compiled: 2013-10-07T06:28Z by hortonmu from branch-2.2.0
Cluster ID: CID-e47f6b2e-ca1f-4eda-b6f7-dd9d5314c251
Block Pool ID: BP-1227723652-172.11.12.6-1390358446283

Live Datanodes : 2

Node    Transferring Address    Last Contact  Admin State    Configured Capacity (GB)  Used(GB)
 Non DFS Used (GB)  Remaining
master    172.11.12.6:50010         1           In Service         16.15                 0.00
        2.76             13.39
node1     172.11.12.7:50010         1           In Service         16.15                 0.00
        2.75             13.40 

How can I know which node will be called ,master(172.11.12.6)  or node1(172.11.12.6) ?  
Thanks.
---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s)

is intended only for the use of the intended recipient and may be confidential and/or privileged
of 
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication
is 
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or
copying 
is strictly prohibited, and may be unlawful.If you have received this communication in error,please

immediately notify the sender by return e-mail, and delete the original message and all copies
from 
your system. Thank you. 
---------------------------------------------------------------------------------------------------
Mime
View raw message