hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris MacKenzie <stu...@chrismackenziephotography.co.uk>
Subject Re: Partitioning and setup errors
Date Sun, 29 Jun 2014 12:21:22 GMT
HI Vinod,

Thanks for your support. I¹m packaging my application in Eclipse (Kepler) as
a runnable jar. I can get the application to debug there in local mode.

The command line I am using for my app is:

 "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10²

the jar permissions are -"rwxr-xr-x"

The three parameters on the end of the command line are arbitrary commands
from my sequential program which I will use as input when I can get this to
run.

I have downloaded the source and run a 2.4 version of WordCount, it¹s not
getting past the map stage either.

I¹m pasting in the whole job output.
Deleted demo_output
/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
/usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
Start time = 1404044126713
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
parsing not performed. Implement the Tool interface and execute your
application with ToolRunner to remedy this.
14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set.  User
classes may not be found. See Job or Job#setJar(String).
14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process :
0
14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1403953980618_0009
14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not adding
any jar to the list of resources.
14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
application_1403953980618_0009
14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running in
uber mode : false
14/06/29 13:17:19 INFO mapreduce.Job:  map 0% reduce 0%
14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at 
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at 
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more


14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at 
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at 
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more


14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
attempt_1403953980618_0009_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at 
org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextIm
pl.java:210)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class
par.gene.align.concordance.ConcordanceReducer not found
at 
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 8 more


14/06/29 13:17:37 INFO mapreduce.Job:  map 0% reduce 100%
14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed with
state FAILED due to: Task failed task_1403953980618_0009_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1


14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
Job Counters
Failed reduce tasks=4
Launched reduce tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=7786
Total time spent by all reduce tasks (ms)=7786
Total vcore-seconds taken by all reduce tasks=7786
Total megabyte-seconds taken by all reduce tasks=7972864
14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s); retry
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000
MILLISECONDS)
14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=FAILED. Redirecting to job history server
End time = 1404044288031
Elapsed time = 161318 ms
Finished
/usr/local/hadoop-2.4.0/jar_files $
Warm regards.
Chris

From:  Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Sunday, 29 June 2014 00:20
To:  <user@hadoop.apache.org>
Subject:  Re: Partitioning and setup errors

What is happening is the client is not able to pick up the right jar to push
to the cluster. It looks in the class-path for the jar that contains the
class ParallelGeneticAlignment.

How are you packaging your code? How are your running your job - paste the
command line?

+Vinod 

On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
<studio@chrismackenziephotography.co.uk> wrote:

> Hi,
> 
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
> 
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if  I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
> 
> I have several faults On my own computer  Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
> 
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set.  User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
> 
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
> 
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> 
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
> 
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
> 
> 
> job.waitForCompletion(true)
> }
> 
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
> 
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
> 
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
> 
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
> 
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
> 
> Thanks in advance,
> Regards,
> 
> Chris MacKenzie


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.


Mime
View raw message