hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Silvina CaĆ­no Lores <silvi.ca...@gmail.com>
Subject Re: pipes on hadoop 2.2.0 crashes
Date Tue, 17 Dec 2013 09:41:23 GMT
I'm having similar problems with pipes, mostly because of issues with the
native shared libraries that leave the job stuck either at 0%-0% or before
launch (because the resource manager gets stuck as well and crashes).

I found that out by looking at the stderr logs by the way.

Let us know if you progress, good luck!

Best,
Silvina


On 16 December 2013 19:32, Vinod Kumar Vavilapalli
<vinodkv@hortonworks.com>wrote:

> You should navigate to the ResourceManager UI following the link and see
> what is happening on the ResourceManager as well as the application-master.
> Check if any nodes are active first. Then look at ResourceManager and
> NodeManager logs.
>
> +Vinod
>
> On Dec 16, 2013, at 10:29 AM, Mauro Del Rio <mdrio1980@gmail.com> wrote:
>
> I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a
> simple wordcount in c++ with pipes, this time there was no exception, but
> the job didn't finish. This is the output on the shell where I launched
> pipes:
>
> mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc
> -input test.sh -output out
> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:9052
> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:9052
> 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set.  User
> classes may not be found. See Job or Job#setJar(String).
> 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2
> 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is
> deprecated. Instead, use mapreduce.job.user.name
> 13/12/16 18:51:41 INFO Configuration.deprecation:
> mapred.cache.files.filesizes is deprecated. Instead, use
> mapreduce.job.cache.files.filesizes
> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is
> deprecated. Instead, use mapreduce.job.cache.files
> 13/12/16 18:51:41 INFO Configuration.deprecation:
> mapred.pipes.user.inputformat is deprecated. Instead, use
> mapreduce.pipes.inputformat
> 13/12/16 18:51:41 INFO Configuration.deprecation:
> mapred.output.value.class is deprecated. Instead, use
> mapreduce.job.output.value.class
> 13/12/16 18:51:41 INFO Configuration.deprecation:
> mapred.mapoutput.value.class is deprecated. Instead, use
> mapreduce.map.output.value.class
> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is
> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.partitioner
> is deprecated. Instead, use mapreduce.pipes.partitioner
> 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.executable
> is deprecated. Instead, use mapreduce.pipes.executable
> 13/12/16 18:51:41 INFO Configuration.deprecation:
> mapred.cache.files.timestamps is deprecated. Instead, use
> mapreduce.job.cache.files.timestamps
> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.key.class
> is deprecated. Instead, use mapreduce.job.output.key.class
> 13/12/16 18:51:41 INFO Configuration.deprecation:
> mapred.mapoutput.key.class is deprecated. Instead, use
> mapreduce.map.output.key.class
> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir
> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1387213974967_0003
> 13/12/16 18:51:42 INFO mapred.YARNRunner: Job jar is not present. Not
> adding any jar to the list of resources.
> 13/12/16 18:51:42 INFO impl.YarnClientImpl: Submitted application
> application_1387213974967_0003 to ResourceManager at /0.0.0.0:9052
> 13/12/16 18:51:42 INFO mapreduce.Job: The url to track the job:
> http://mauro-VirtualBox:8088/proxy/application_1387213974967_0003/<http://mauro-virtualbox:8088/proxy/application_1387213974967_0003/>
> 13/12/16 18:51:42 INFO mapreduce.Job: Running job: job_1387213974967_0003
>
>
> The job status from bin/mapred job -list  is PREP. I didn't find any
> interesting information in logs file.
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
View raw message