hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Where is hdfs result ?
Date Mon, 23 Jun 2014 11:29:31 GMT
The pi job does not leave any HDFS data. You can see its result on the
console: "Estimated value of Pi is 3.14127500000000000000".

What are you trying to really test or do here? If you want data on
HDFS, you can load it up via 'hadoop fs -put', or generate it with
teragen, etc.?

On Mon, Jun 23, 2014 at 12:06 PM, EdwardKing <zhangsc@neusoft.com> wrote:
>  I use hadoop 2.2.0 under Redhat Linux,I run hadoop mapreduce examples like
> follows:
>
> [yarn@localhost sbin]$ hadoop jar
> /opt/yarn/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar
> pi
> -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory
> -libjars
> /opt/yarn/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar
> 16 10000
> Number of Maps  = 16
> Samples per Map = 10000
> Wrote input for Map #0
> Wrote input for Map #1
> Wrote input for Map #2
> Wrote input for Map #3
> Wrote input for Map #4
> Wrote input for Map #5
> Wrote input for Map #6
> Wrote input for Map #7
> Wrote input for Map #8
> Wrote input for Map #9
> Wrote input for Map #10
> Wrote input for Map #11
> Wrote input for Map #12
> Wrote input for Map #13
> Wrote input for Map #14
> Wrote input for Map #15
> Starting Job
> 14/06/22 19:57:58 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/06/22 19:57:59 INFO input.FileInputFormat: Total input paths to process :
> 16
> 14/06/22 19:57:59 INFO mapreduce.JobSubmitter: number of splits:16
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.job.classpath.files
> is deprecated. Instead, use mapreduce.job.classpath.files
> 14/06/22 19:57:59 INFO Configuration.deprecation: user.name is deprecated.
> Instead, use mapreduce.job.user.name
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 14/06/22 19:57:59 INFO Configuration.deprecation:
> mapred.cache.files.filesizes is deprecated. Instead, use
> mapreduce.job.cache.files.filesizes
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.cache.files is
> deprecated. Instead, use mapreduce.job.cache.files
> 14/06/22 19:57:59 INFO Configuration.deprecation:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.output.value.class
> is deprecated. Instead, use mapreduce.job.output.value.class
> 14/06/22 19:57:59 INFO Configuration.deprecation:
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
> 14/06/22 19:57:59 INFO Configuration.deprecation:
> mapred.used.genericoptionsparser is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.job.name is
> deprecated. Instead, use mapreduce.job.name
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapreduce.reduce.class is
> deprecated. Instead, use mapreduce.job.reduce.class
> 14/06/22 19:57:59 INFO Configuration.deprecation:
> mapreduce.inputformat.class is deprecated. Instead, use
> mapreduce.job.inputformat.class
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 14/06/22 19:57:59 INFO Configuration.deprecation:
> mapreduce.outputformat.class is deprecated. Instead, use
> mapreduce.job.outputformat.class
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 14/06/22 19:57:59 INFO Configuration.deprecation:
> mapred.cache.files.timestamps is deprecated. Instead, use
> mapreduce.job.cache.files.timestamps
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 14/06/22 19:57:59 INFO Configuration.deprecation: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 14/06/22 19:58:00 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1403492007236_0001
> 14/06/22 19:58:01 INFO impl.YarnClientImpl: Submitted application
> application_1403492007236_0001 to ResourceManager at /0.0.0.0:8032
> 14/06/22 19:58:01 INFO mapreduce.Job: The url to track the job:
> http://localhost:8088/proxy/application_1403492007236_0001/
> 14/06/22 19:58:01 INFO mapreduce.Job: Running job: job_1403492007236_0001
> 14/06/22 19:58:22 INFO mapreduce.Job: Job job_1403492007236_0001 running in
> uber mode : false
> 14/06/22 19:58:22 INFO mapreduce.Job:  map 0% reduce 0%
> 14/06/22 20:01:45 INFO mapreduce.Job:  map 6% reduce 0%
> 14/06/22 20:01:52 INFO mapreduce.Job:  map 13% reduce 0%
> 14/06/22 20:01:53 INFO mapreduce.Job:  map 19% reduce 0%
> 14/06/22 20:01:56 INFO mapreduce.Job:  map 38% reduce 0%
> 14/06/22 20:02:36 INFO mapreduce.Job:  map 50% reduce 15%
> 14/06/22 20:02:37 INFO mapreduce.Job:  map 69% reduce 15%
> 14/06/22 20:02:39 INFO mapreduce.Job:  map 69% reduce 23%
> 14/06/22 20:03:05 INFO mapreduce.Job:  map 94% reduce 23%
> 14/06/22 20:03:06 INFO mapreduce.Job:  map 100% reduce 31%
> 14/06/22 20:03:08 INFO mapreduce.Job:  map 100% reduce 100%
> 14/06/22 20:03:09 INFO mapreduce.Job: Job job_1403492007236_0001 completed
> successfully
> 14/06/22 20:03:09 INFO mapreduce.Job: Counters: 43
>  File System Counters
>   FILE: Number of bytes read=358
>   FILE: Number of bytes written=1387065
>   FILE: Number of read operations=0
>   FILE: Number of large read operations=0
>   FILE: Number of write operations=0
>   HDFS: Number of bytes read=4230
>   HDFS: Number of bytes written=215
>   HDFS: Number of read operations=67
>   HDFS: Number of large read operations=0
>   HDFS: Number of write operations=3
>  Job Counters
>   Launched map tasks=16
>   Launched reduce tasks=1
>   Data-local map tasks=16
>   Total time spent by all maps in occupied slots (ms)=1587720
>   Total time spent by all reduces in occupied slots (ms)=71305
>  Map-Reduce Framework
>   Map input records=16
>   Map output records=32
>   Map output bytes=288
>   Map output materialized bytes=448
>   Input split bytes=2342
>   Combine input records=0
>   Combine output records=0
>   Reduce input groups=2
>   Reduce shuffle bytes=448
>   Reduce input records=32
>   Reduce output records=0
>   Spilled Records=64
>   Shuffled Maps =16
>   Failed Shuffles=0
>   Merged Map outputs=16
>   GC time elapsed (ms)=45172
>   CPU time spent (ms)=107260
>   Physical memory (bytes) snapshot=2563190784
>   Virtual memory (bytes) snapshot=6111346688
>   Total committed heap usage (bytes)=1955135488
>  Shuffle Errors
>   BAD_ID=0
>   CONNECTION=0
>   IO_ERROR=0
>   WRONG_LENGTH=0
>   WRONG_MAP=0
>   WRONG_REDUCE=0
>  File Input Format Counters
>   Bytes Read=1888
>  File Output Format Counters
>   Bytes Written=97
> Job Finished in 311.613 seconds
> Estimated value of Pi is 3.14127500000000000000
>
> Then I check hdfs like follows:
> [yarn@localhost sbin]$ hdfs dfs -ls
> [yarn@localhost sbin]
>
> Why hdfs don't show any information? How to do it? Thanks
>
>
>
> ---------------------------------------------------------------------------------------------------
> Confidentiality Notice: The information contained in this e-mail and any
> accompanying attachment(s)
> is intended only for the use of the intended recipient and may be
> confidential and/or privileged of
> Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader
> of this communication is
> not the intended recipient, unauthorized use, forwarding, printing,
> storing, disclosure or copying
> is strictly prohibited, and may be unlawful.If you have received this
> communication in error,please
> immediately notify the sender by return e-mail, and delete the original
> message and all copies from
> your system. Thank you.
> ---------------------------------------------------------------------------------------------------



-- 
Harsh J

Mime
View raw message